Validation
Test vectors
Accredited laboratories have to validate the software they use for calibration results (ISO/IEC 17025 clause 7.11). These are the reference cases every Metrology Bench calculator is tested against on every change: an input, the expected output, how close the answer must be, and where the expected value comes from. Run your own spreadsheet or software against them.
Each vector: inputs, expected value, allowed tolerance, and where the expected value comes from. Type K expected values are NIST SRD 60 table entries (rounded by NIST to 0.001 mV / whole °C). RTD expected values are from the Callendar–Van Dusen equation with Pt385 coefficients, cross-checked against IEC 60751 table values (rounded to 0.01 Ω). Pressure and torque expected values are derived from SI definitions as tabulated in NIST SP 811.
| ID | Function | Inputs | Expected | Tolerance | Why it matters | Source |
|---|---|---|---|---|---|---|
| TCK-F-01 | tcEmf | type = K, t_C = -200 | -5.891 mV | ± 0.0005 | Low end of inverse range | NIST SRD 60 type K table, -200 °C |
| TCK-F-02 | tcEmf | type = K, t_C = -100 | -3.554 mV | ± 0.0005 | Negative range, mid | NIST SRD 60 type K table, -100 °C |
| TCK-F-03 | tcEmf | type = K, t_C = -50 | -1.889 mV | ± 0.0005 | Negative range, near zero | NIST SRD 60 type K table, -50 °C |
| TCK-F-04 | tcEmf | type = K, t_C = 0 | 0 mV | ± 0.0005 | Segment boundary; exponential term starts | NIST SRD 60 type K table, 0 °C |
| TCK-F-05 | tcEmf | type = K, t_C = 25 | 1 mV | ± 0.0005 | Room temperature | NIST SRD 60 type K table, 25 °C |
| TCK-F-06 | tcEmf | type = K, t_C = 100 | 4.096 mV | ± 0.0005 | Common check point | NIST SRD 60 type K table, 100 °C |
| TCK-F-07 | tcEmf | type = K, t_C = 200 | 8.138 mV | ± 0.0005 | Positive range, mid | NIST SRD 60 type K table, 200 °C |
| TCK-F-08 | tcEmf | type = K, t_C = 500 | 20.644 mV | ± 0.0005 | Inverse segment boundary | NIST SRD 60 type K table, 500 °C |
| TCK-F-09 | tcEmf | type = K, t_C = 1000 | 41.276 mV | ± 0.0005 | High range | NIST SRD 60 type K table, 1000 °C |
| TCK-F-10 | tcEmf | type = K, t_C = 1372 | 54.886 mV | ± 0.0005 | Top of range | NIST SRD 60 type K table, 1372 °C |
| TCK-I-01 | tcTemperature | type = K, emf_mV = -5.891 | -200 °C | ± 0.08 | Bottom of inverse range. Tolerance = NIST inverse error band plus the 0.0005 mV table rounding at low sensitivity | NIST SRD 60 type K table, -200 °C |
| TCK-I-02 | tcTemperature | type = K, emf_mV = -3.554 | -100 °C | ± 0.06 | Negative inverse segment | NIST SRD 60 type K table, -100 °C |
| TCK-I-03 | tcTemperature | type = K, emf_mV = 0 | 0 °C | ± 0.06 | Inverse segment boundary | NIST SRD 60 type K table, 0 °C |
| TCK-I-04 | tcTemperature | type = K, emf_mV = 4.096 | 100 °C | ± 0.06 | Common check point | NIST SRD 60 type K table, 100 °C |
| TCK-I-05 | tcTemperature | type = K, emf_mV = 20.644 | 500 °C | ± 0.08 | Inverse segment boundary (upper segment used) | NIST SRD 60 type K table, 500 °C |
| TCK-I-06 | tcTemperature | type = K, emf_mV = 41.276 | 1000 °C | ± 0.08 | High range | NIST SRD 60 type K table, 1000 °C |
| TCK-I-07 | tcTemperature | type = K, emf_mV = 54.886 | 1372 °C | ± 0.08 | Top of range | NIST SRD 60 type K table, 1372 °C |
| RTD-F-01 | rtdResistance | t_C = -200, r0 = 100 | 18.52 Ω | ± 0.005 | Bottom of range; C term active | CvD Pt385; cross-check IEC 60751 table |
| RTD-F-02 | rtdResistance | t_C = -100, r0 = 100 | 60.26 Ω | ± 0.005 | C term active | CvD Pt385; cross-check IEC 60751 table |
| RTD-F-03 | rtdResistance | t_C = 0, r0 = 100 | 100 Ω | ± 0.00005 | Equation changes form at 0 °C | Definition of R0 |
| RTD-F-04 | rtdResistance | t_C = 100, r0 = 100 | 138.5055 Ω | ± 0.00005 | Common check point | CvD Pt385, exact arithmetic: 100(1 + 0.39083 − 0.005775) |
| RTD-F-05 | rtdResistance | t_C = 200, r0 = 100 | 175.86 Ω | ± 0.005 | Positive range, mid | CvD Pt385; cross-check IEC 60751 table |
| RTD-F-06 | rtdResistance | t_C = 500, r0 = 100 | 280.98 Ω | ± 0.005 | High range | CvD Pt385; cross-check IEC 60751 table |
| RTD-F-07 | rtdResistance | t_C = 850, r0 = 100 | 390.48 Ω | ± 0.005 | Top of range | CvD Pt385; cross-check IEC 60751 table |
| RTD-F-08 | rtdResistance | t_C = 100, r0 = 1000 | 1385.055 Ω | ± 0.0005 | R0 scaling | CvD Pt385 scaled to R0 = 1000 Ω (Pt1000) |
| RTD-I-01 | rtdTemperature | r_ohm = 138.5055, r0 = 100 | 100 °C | ± 0.0001 | Exact quadratic inverse | Inverse of RTD-F-04 |
| RTD-I-02 | rtdTemperature | r_ohm = 60.2558, r0 = 100 | -100 °C | ± 0.001 | Iterative inverse below 0 °C | Inverse of CvD Pt385 at −100 °C (60.2558 Ω) |
| RTD-I-03 | rtdTemperature | r_ohm = 100, r0 = 100 | 0 °C | ± 0.0001 | Boundary between the two forms | Definition of R0 |
| RANGE-01 | tcEmf | type = K, t_C = 1373 | OutOfRangeError | exact | Must refuse, not extrapolate | NIST type K range −270 to 1372 °C |
| RANGE-02 | tcTemperature | type = K, emf_mV = 55 | OutOfRangeError | exact | Must refuse, not extrapolate | NIST type K inverse range −5.891 to 54.886 mV |
| RANGE-03 | rtdResistance | t_C = 851, r0 = 100 | OutOfRangeError | exact | Must refuse, not extrapolate | CvD valid range −200 to 850 °C |
| ROUND-01 | roundDecimals | x = 2.675, places = 2, mode = half-up | 2.68 | exact | Binary float would give 2.67; must not | Decimal rounding rule |
| ROUND-02 | roundDecimals | x = 2.45, places = 1, mode = half-even | 2.4 | exact | Tie goes to even digit | Round half to even |
| ROUND-03 | roundDecimals | x = 2.55, places = 1, mode = half-even | 2.6 | exact | Tie goes to even digit | Round half to even |
| ROUND-04 | roundDecimals | x = -2.5, places = 0, mode = half-up | -3 | exact | Negative ties | Round half away from zero |
| ROUND-05 | roundSigFigs | x = 0.0012345, digits = 3, mode = half-up | 0.00123 | exact | Leading zeros are not significant | Significant figures |
| ROUND-06 | roundSigFigs | x = 9.996, digits = 3, mode = half-up | 10.0 | exact | Carry into a new digit | Significant figures |
| PRS-01 | convertPressure | value = 100, from = psi, to = kPa | 689.4757293 kPa | ± 5e-8 | Most common pressure conversion | NIST SP 811: 1 psi = 6.894 757 293 kPa (derived from the definitions of the pound, inch and standard gravity) |
| PRS-02 | convertPressure | value = 1, from = atm, to = psi | 14.69594878 psi | ± 5e-9 | Standard atmosphere | 101 325 Pa ÷ 6 894.757 293 Pa |
| PRS-03 | convertPressure | value = 1, from = bar, to = psi | 14.50377377 psi | ± 5e-9 | Metric to imperial | 100 000 Pa ÷ 6 894.757 293 Pa |
| PRS-04 | convertPressure | value = 1, from = inHg, to = Pa | 3386.38864 Pa | ± 0.000005 | Manometric unit, conventional mercury density | Conventional: 0.0254 m × 13 595.1 kg/m³ × 9.806 65 m/s² |
| PRS-05 | convertPressure | value = 1, from = inH2O, to = Pa | 249.08891 Pa | ± 0.000005 | Manometric unit, conventional water density | Conventional: 0.0254 m × 1000 kg/m³ × 9.806 65 m/s² |
| PRS-06 | convertPressure | value = 760, from = torr, to = atm | 1 atm | ± 1e-12 | Exact definition | Definition: 760 torr = 1 atm |
| TRQ-01 | convertTorque | value = 1, from = ftlb, to = Nm | 1.3558179483 N·m | ± 5e-11 | Most common torque conversion | 0.453 592 37 kg × 9.806 65 m/s² × 0.3048 m (NIST SP 811) |
| TRQ-02 | convertTorque | value = 1, from = Nm, to = inlb | 8.850745791 in·lbf | ± 5e-10 | Small-tool torque | 1 ÷ 0.112 984 829 N·m (NIST SP 811) |
| TRQ-03 | convertTorque | value = 1, from = inoz, to = cNm | 0.7061551814 cN·m | ± 5e-11 | Screwdriver-range unit | (0.453 592 37 ÷ 16) kg × 9.806 65 m/s² × 0.0254 m |
| TRQ-04 | convertTorque | value = 12, from = inlb, to = ftlb | 1 ft·lbf | ± 1e-12 | Exact relationship | Definition: 12 in = 1 ft |
| TRQ-05 | torqueExtensionSetting | target = 100, wrenchLength = 18, extension = 3 | 85.71428571 same as target | ± 5e-9 | Inline extension correction | setting = target × L ÷ (L + E) = 100 × 18 ÷ 21 |
| TRQ-06 | torqueExtensionSetting | target = 100, wrenchLength = 18, extension = 0 | 100 same as target | ± 1e-12 | Zero-extension edge case | No extension: setting equals target |
Found a case we should add, or a value you disagree with? Tell us.