Accuracy, precision, and resolution are three distinct properties of a measuring instrument — and confusing them leads to poor calibration decisions and misread measurement results. Accuracy describes how close a measurement is to the true value. Precision describes how consistently an instrument produces the same reading under repeated conditions. Resolution is the smallest increment of change the instrument can detect and display. An instrument can be precise without being accurate, accurate on average while being imprecise, and high-resolution while being neither accurate nor precise. Understanding the difference shapes how you select instruments, set tolerances, interpret calibration certificates, and respond to quality system audits.
Key Takeaways
- Accuracy is closeness to the true value; precision is repeatability; resolution is the smallest detectable increment
- Calibration directly addresses accuracy — it identifies and corrects systematic error (bias); it does not improve precision or resolution
- A high-resolution display can create false confidence if the instrument’s accuracy or precision is poor
- Gauge R&R studies measure precision components; calibration measures accuracy — they answer different questions and cannot substitute for each other
What Is Measurement Accuracy?
Accuracy in measurement is the degree of agreement between an instrument’s reading and the true value of the quantity being measured. A pressure gauge that reads 100.3 psi when the true pressure is 100.0 psi has a positive error of 0.3 psi — its accuracy at that point is 0.3% of reading. The closer the instrument’s reading to the true value across its full measurement range, the higher its accuracy.
Accuracy is always stated relative to a reference. The “true value” in calibration is defined by the reference standard used to check the instrument. This is why traceability matters: the accuracy you can claim for your instrument depends on the accuracy of the reference used to establish it. Working standards and reference standards form the traceability chain that connects your instrument’s accuracy claim back to national and international measurement standards.
Accuracy can degrade over time through calibration drift — the gradual shift of an instrument’s reading away from the true value. Calibration exists specifically to detect and correct accuracy errors. When a calibration technician adjusts an instrument after finding it out of tolerance, they are correcting its accuracy. The instrument’s precision and resolution are unchanged by that adjustment.
What Is Measurement Precision?
Precision is the degree to which repeated measurements of the same quantity under the same conditions produce the same result. A precise instrument gives consistent readings — but those readings may be consistently wrong. Precision has two components: repeatability (precision under identical conditions by the same operator in a short time frame) and reproducibility (precision across different operators, instruments, or time periods).
Precision is measured through statistical analysis of repeated readings, not through calibration. A Gauge Repeatability and Reproducibility study — Gauge R&R — is the standard method for quantifying a measurement system’s precision components. Gauge R&R and calibration address different questions: calibration asks “is this instrument accurate?”, while Gauge R&R asks “is this measurement system consistent enough to detect the variation we care about?”
Precision problems are often caused by instrument design limitations, poor measurement technique, or environmental variation — not by accuracy errors that calibration can fix. An imprecise instrument that is calibrated and adjusted will still produce inconsistent readings after calibration, because the inconsistency comes from variability in the measurement process itself, not from a systematic offset in accuracy.
What Is Measurement Resolution?
Resolution is the smallest increment of change that an instrument can detect and display. A digital thermometer with 0.1°C display resolution cannot meaningfully report temperature differences smaller than 0.1°C — even if the true temperature changes by 0.05°C, the display will not change. A dial indicator with 0.001″ graduation cannot detect displacements smaller than its graduation, regardless of how accurate or precise the instrument is in other respects.
Resolution is a fundamental property of the instrument’s design and display. It cannot be changed by calibration. A higher-resolution display is not inherently more accurate — it simply shows more decimal places. An instrument with 0.001°C display resolution that is out of calibration by 0.5°C is far less useful than an instrument with 0.1°C resolution that is well within calibration.
Resolution matters in calibration when selecting reference standards. The calibration reference must have sufficient resolution to meaningfully detect errors in the instrument being calibrated. A reference that displays only one more decimal place than the instrument under calibration offers limited ability to characterize measurement errors at fine levels — this is one reason the 4:1 Test Uncertainty Ratio guideline exists.
Why Calibration Addresses Accuracy but Not Precision or Resolution
Calibration identifies the systematic error (bias) in an instrument’s reading by comparing it against a reference standard at defined calibration points. If the instrument reads 0.3% high, the calibration record documents that error, and the technician either adjusts the instrument or notes the correction factor to apply to future readings. The resulting calibration certificate confirms the instrument’s accuracy within stated tolerances.
Nothing in this process changes how consistently the instrument reads (precision) or how finely it can resolve differences (resolution). Those are properties of the instrument’s physical and electronic design. Calibration cannot make a dial indicator read to 0.0001″ if it was built with 0.001″ graduation. It cannot make a noisy sensor produce repeatable readings if the noise is electrical or mechanical in origin.
This distinction becomes important when responding to measurement problems. If a quality complaint traces back to inconsistent measurements, the instinct may be to “send it out for calibration.” But if the problem is precision — scattered readings around the correct value — calibration won’t fix it. The investigation needs to look at the measurement system as a whole: the instrument, the operator technique, the fixture, the environment. That’s the scope of a Gauge R&R study, not a calibration.
When reviewing calibration certificates, the expanded uncertainty reported reflects the uncertainty in the accuracy determination — it does not characterize the instrument’s precision in your application. Both are important; neither substitutes for the other.
For calibration services that provide clear accuracy and uncertainty documentation on every certificate, Micro Precision’s calibration services are built to ISO/IEC 17025 requirements. Request a quote.
Frequently Asked Questions
Accuracy is hitting the bullseye — your measurement is close to the true value. Precision is grouping your shots tightly — your repeated measurements are consistent with each other. You can be precise without being accurate (consistently wrong in the same direction), and accurate on average without being precise (scattered around the right value). Calibration improves accuracy. Better technique and instrument quality improve precision.
Calibration directly improves accuracy by identifying and correcting systematic errors. It does not improve precision — the repeatability and reproducibility of the measurement process are determined by the instrument’s design, the measurement environment, and the operator’s technique. A Gauge R&R study, not calibration, is the right tool for evaluating and improving measurement precision.
A precise but inaccurate instrument consistently produces the wrong answer — and does so reliably. Every measurement will be systematically off by the same amount, and without calibration there is no way to know by how much. This is arguably more dangerous than an imprecise instrument, because the consistent readings give false confidence. Calibration of precise instruments is just as important as calibration of less precise ones.
Not necessarily. Higher resolution only adds value when the instrument has the accuracy and precision to support it. A thermometer with 0.001°C display resolution but 0.5°C accuracy is no more useful than a 0.1°C thermometer at the same accuracy. Resolution should match the application requirement, with accuracy and precision appropriate for the tolerance being measured.
Resolution contributes to calibration uncertainty as a quantization error. When a digital instrument displays in increments of 0.1, the reading could be off by up to ±0.05 units from rounding alone. This quantization component is included in the calibration uncertainty budget, especially when the instrument’s resolution is large relative to its tolerance.
Accuracy describes the closeness of a measurement result to the true value. Uncertainty quantifies the range within which the true value is expected to fall, given all known sources of error. A calibration certificate reports expanded uncertainty — combining the reference standard’s accuracy, resolution effects, environmental contributions, and other factors. Accuracy is the result; uncertainty is the statement of how well that result is known.