Controlling resolution for results generated by instrument evaluation
The Device Under Test (DUT) resolution is determined by the following rules:
- If the instrument FSC Nominal field contains a value, use it to determine the resolution.
Example
5700 400 350.0mV 2.9U 60H 2W
will give one significant digit in the results.
IEEE [T30000]OUT 2Mv, 10Hz; OPER;*OPC?[I]
5790 2.2 2.00000mV 0.305% 10H I1 2W
- If the instrument FSC Nominal field does not contain a value format control is not possible. An instrument evaluation test without a Nominal field value must be broken into an instrument setup and a MEMC or MEMCX FSC to properly control the resolution (See MEMC below)
Controlling Resolution for Results Generated by the MEMC/MEMCX FSC
The resolution is determined by the following rules:
- If the MEMC or MEMCX Nominal field contains a value, use it to determine the resolution. This case is not typically used. If the MEMC Nominal field contains a value, it is taken as the DUT Indicated value. A constant DUT Indicated value only makes sense for evaluation of a source DUT. Source DUTs are typically tested using instrument evaluations such as the 5790 statement in the example above. However, this case must be used if a User-configured Device (standard for which there is no FSC) is used to evaluate a source DUT.
Example
IEEE AP-12DB, SP84
IEEE [@436][I]
MEMC -12.0D 1.0U
- Otherwise, if there is a prior ACC statement, use the ACC Nominal field value to determine the resolution. This is the case when the test units are not the same as the units of the standard.
Example
5700 2V 60H S 2W
DISP Press the MIN MAX button to display minimum values
ACC -2.828Vp 75P% 7e-6U
MEMI Enter DUT reading in volts peak
MEMCX Vp 0.045U
- Otherwise, if there was a prior setup statement, but no prior ACC statement, use the Nominal field value of the setup statement to determine the resolution. This is the typical case for closed-loop meter procedures or non-closed loop procedures when the Nominal field of the instrument FSC does not contain a value. The value is taken from memory register MEM.
Example
5700 1.00000V 200H S 2W
IEEE ?[I]
MEMCX 2 v 0.00150U 200H
In this case the resolution is 5 significant digits
MATH MEM = 350
5700 mV 60H S 2W
MEMI Enter the DUT reading in microfarads
MEMCX 1 uF 1%
In this case the resolution is no significant digits
- Otherwise, if the MEMC Tolerance is specified in units of the Nominal “U”, use it to determine the resolution. This is the case when a test is performed with no input connected to the DUT.
Example
DISP Disconnect the 5700A from the DUT.
DISP Press uF
MATH MEM1 = 0.0
MEMI Enter DUT reading in microfarads
MEMCX 1 uF 0.001U
- Finally, if none of the above conditions are met, MET/CAL will format the results with up to 10 digits to the right of the decimal point to preserve non-zero digits. However, trailing zeros to the right of the last non-zero digit will be stripped off.