Accuracy spec → tolerance limits

Turn a manufacturer accuracy spec into tolerance limits at a test point: % of reading, % of range, counts at a resolution, ppm, % of full scale, absolute terms and “whichever is greater” floors. It shows how it read each term, so you can check the interpretation, and converts the half-width to a standard uncertainty.

Accuracy spec · tolerance at a test point

v0.1.0 · GUM §4.3

Reading or nominalUnitRange (for % or ppm of range)Full scale (for % FS)Resolution, one count

Spec terms, all in the reading’s unit; ± values from the datasheet, blank if not used% of reading% of range% of full scaleppm of readingppm of rangeCounts (digits)Absolute term (+)Floor, whichever is greaterSpec stated as

Tolerance limits4.9986 to 5.0014 Va = ± 0.0014 V · u = 0.000808 V (a/√3)

How the spec was read

0.02 % of reading (5 V) = 0.001 V
4 counts × 0.0001 V resolution = 0.0004 V
total 0.0014 V
Limits: 5 ± 0.0014 V = 4.9986 to 5.0014 V
Standard uncertainty: spec read as a bound, rectangular, u = a/√3 = 0.0008082903769 V

a = sum of terms (or the largest, for “whichever is greater”) · limits = reading ± a · counts × resolution

ShowFiguresRounding

Show full precision

How this is calculated

Equation: a = Σ terms (or the largest group total for "whichever is greater"); % of X = p/100 × X; ppm of X = q × 10⁻⁶ × X; counts = n × resolution; limits = reading ± a; u = a/√3 (rectangular) or a/k (stated coverage factor)
Source: Term arithmetic from the definitions of each spec term (no standard defines spec notation; read the manufacturer’s datasheet for the basis, period and confidence). Standard uncertainty per JCGM 100:2008 (GUM) §4.3.7 (rectangular bounds) and §4.3.3 (stated coverage factor).
Valid range: Reading within the range (or the range’s stated over-range); range, full scale and resolution greater than zero; term values zero or more
Rounding: Limits and half-width shown to 10 significant figures with trailing zeros dropped; uncertainty to 3 significant figures by default, rounded half up.
Version: v0.1.0 · changelog

Calculated value, not a certified result. Accredited laboratories must validate any software they use (ISO/IEC 17025 clause 7.11).

Test vectors for this tool

The reference cases this calculator is tested against on every change. Use them to check your own software. All test vectors · Download JSON

ID	Inputs	Expected	Tolerance	Source
SPEC-01	reading = 5, resolution = 0.0001, terms = [{"type":"percent","of":"reading","value":0.02},{"type":"counts","value":4}], distribution = {"kind":"rectangular"}, unit = V, output = halfWidth	0.0014 V	± 1e-15	Keysight Knowledge Center, "What is the relationship between the 'Count' and the accuracy of U3401A or U3402A Digital Multimeter?" (https://docs.keysight.com/kkbopen/what-is-the-relationship-between-the-count-and-the-accuracy-of-u3401a-or-u3402a-digital-multimeter-588280703.html): 0.02 % of reading + 4 counts at 5 V, 100 µV resolution = ±0.0014 V. Spec confirmed in Keysight data sheet 5990-3970EN (Aug 2017), U3401A 5 V DC range: 100 µV resolution, ±(0.02 % of reading + 4 counts), 1 year, 23 °C ± 5 °C. The article itself sits behind a bot check and has not been opened; confirm it in a browser.
SPEC-02	reading = 5, resolution = 0.0001, terms = [{"type":"percent","of":"reading","value":0.02},{"type":"counts","value":4}], distribution = {"kind":"rectangular"}, unit = V, output = lower	4.9986 V	± 1e-12	Keysight Knowledge Center, "What is the relationship between the 'Count' and the accuracy of U3401A or U3402A Digital Multimeter?" (https://docs.keysight.com/kkbopen/what-is-the-relationship-between-the-count-and-the-accuracy-of-u3401a-or-u3402a-digital-multimeter-588280703.html): 0.02 % of reading + 4 counts at 5 V, 100 µV resolution = ±0.0014 V. Spec confirmed in Keysight data sheet 5990-3970EN (Aug 2017), U3401A 5 V DC range: 100 µV resolution, ±(0.02 % of reading + 4 counts), 1 year, 23 °C ± 5 °C. The article itself sits behind a bot check and has not been opened; confirm it in a browser.
SPEC-03	reading = 5, resolution = 0.0001, terms = [{"type":"percent","of":"reading","value":0.02},{"type":"counts","value":4}], distribution = {"kind":"rectangular"}, unit = V, output = upper	5.0014 V	± 1e-12	Keysight Knowledge Center, "What is the relationship between the 'Count' and the accuracy of U3401A or U3402A Digital Multimeter?" (https://docs.keysight.com/kkbopen/what-is-the-relationship-between-the-count-and-the-accuracy-of-u3401a-or-u3402a-digital-multimeter-588280703.html): 0.02 % of reading + 4 counts at 5 V, 100 µV resolution = ±0.0014 V. Spec confirmed in Keysight data sheet 5990-3970EN (Aug 2017), U3401A 5 V DC range: 100 µV resolution, ±(0.02 % of reading + 4 counts), 1 year, 23 °C ± 5 °C. The article itself sits behind a bot check and has not been opened; confirm it in a browser.
SPEC-04	reading = 5, resolution = 0.0001, terms = [{"type":"percent","of":"reading","value":0.02},{"type":"counts","value":4}], distribution = {"kind":"rectangular"}, unit = V, output = standardUncertainty	0.0008082903768654761 V	± 1e-15	Hand derivation from the term definitions and JCGM 100:2008 §4.3.7 (bounds, rectangular)
SPEC-05	reading = 5, resolution = 0.0001, terms = [{"type":"percent","of":"reading","value":0.02},{"type":"counts","value":4}], distribution = {"kind":"normal","k":2,"confidence":"95 %"}, unit = V, output = standardUncertainty	0.0007 V	± 1e-15	Hand derivation from the term definitions and JCGM 100:2008 §4.3.3 (stated coverage factor)
SPEC-06	reading = -5, resolution = 0.0001, terms = [{"type":"percent","of":"reading","value":0.02},{"type":"counts","value":4}], distribution = {"kind":"rectangular"}, unit = V, output = lower	-5.0014 V	± 1e-12	Hand derivation from the term definitions
SPEC-07	reading = 40, resolution = 0.001, terms = [{"type":"percent","of":"reading","value":1},{"type":"counts","value":4}], distribution = {"kind":"rectangular"}, unit = V, output = lower	39.596 V	± 1e-12	Hand derivation from the term definitions
SPEC-08	reading = 4, range = 6, resolution = 0.001, terms = [{"type":"percent","of":"reading","value":0.5},{"type":"counts","value":2}], distribution = {"kind":"rectangular"}, unit = V, output = upper	4.022 V	± 1e-12	Hand derivation from the term definitions
SPEC-09	reading = 1, range = 10, terms = [{"type":"percent","of":"reading","value":0.0035},{"type":"percent","of":"range","value":0.0005}], distribution = {"kind":"rectangular"}, unit = V, output = halfWidth	0.000085 V	± 1e-18	Hand derivation from the term definitions
SPEC-10	reading = 1, range = 10, terms = [{"type":"ppm","of":"reading","value":20},{"type":"ppm","of":"range","value":5}], distribution = {"kind":"rectangular"}, unit = V, output = halfWidth	0.00007 V	± 1e-18	Hand derivation from the term definitions
SPEC-11	reading = 100, fullScale = 300, terms = [{"type":"percent","of":"full_scale","value":0.25}], distribution = {"kind":"rectangular"}, unit = psi, output = halfWidth	0.75 psi	± 1e-14	Hand derivation from the term definitions
SPEC-12	reading = 100, terms = [{"type":"percent","of":"reading","value":0.05},{"type":"absolute","value":0.002}], distribution = {"kind":"rectangular"}, unit = mV, output = halfWidth	0.052 mV	± 1e-15	Hand derivation from the term definitions
SPEC-13	reading = 5, terms = [{"type":"percent","of":"reading","value":1}], orGreaterOf = [[{"type":"absolute","value":0.1}]], distribution = {"kind":"rectangular"}, unit = V, output = halfWidth	0.1 V	± 1e-15	Hand derivation from the term definitions
SPEC-14	reading = 20, terms = [{"type":"percent","of":"reading","value":1}], orGreaterOf = [[{"type":"absolute","value":0.1}]], distribution = {"kind":"rectangular"}, unit = V, output = halfWidth	0.2 V	± 1e-15	Hand derivation from the term definitions
SPEC-15	reading = 5, terms = [{"type":"percent","of":"reading","value":0.02},{"type":"counts","value":4}], distribution = {"kind":"rectangular"}, unit = V, output = halfWidth	SpecInputError	exact	Metrology Bench rule: never guess a missing piece
SPEC-16	reading = 5, terms = [{"type":"percent","value":0.02}], distribution = {"kind":"rectangular"}, unit = V, output = halfWidth	SpecInputError	exact	Metrology Bench rule: never guess a missing piece
SPEC-17	reading = 5000, range = 10, terms = [{"type":"percent","of":"reading","value":0.02}], distribution = {"kind":"rectangular"}, unit = V, output = halfWidth	OutOfRangeError	exact	Metrology Bench rule: refuse, don’t extrapolate
SPEC-18	reading = 5, resolution = 0.0001, terms = [{"type":"percent","of":"reading","value":0.02},{"type":"counts","value":4}], distribution = {"kind":"normal"}, unit = V, output = standardUncertainty	SpecInputError	exact	Metrology Bench rule: never guess a missing piece

Put this calculator on your site

Free for labs, schools and anyone else. Paste this into your page.

<iframe src="https://metrologybench.com/embed/spec-tolerance" title="Accuracy spec → tolerance limits · Metrology Bench" width="100%" height="970" style="border:0;max-width:640px" loading="lazy"></iframe>

Preview the embed

Ask your AI

Connect metrology-toolbox-mcp and your assistant runs this exact calculation, with the source in the answer.

Set it up