When an object detection model outputs a confidence score of 0.85 on a bounding box, most teams interpret that as a high-quality detection that doesn't need human review. That interpretation is often wrong, and it's wrong in a way that silently corrupts your annotation queue prioritization.
Confidence scores measure how certain a model is about its own prediction, not how correct the prediction is. A poorly calibrated model can be very certain about predictions that are frequently wrong. Unless you've explicitly measured the relationship between your model's confidence scores and its actual accuracy rate, you don't know whether 0.85 means "85% chance this label is correct" or "the model is very confident in a prediction that's right about 60% of the time."
What calibration actually measures
Calibration is the alignment between a model's expressed confidence and its empirical accuracy. A perfectly calibrated model that outputs 0.7 confidence on a set of predictions is correct on exactly 70% of those predictions. Most neural networks are not well-calibrated out of the box. They tend to be overconfident, particularly on classes they've seen frequently during training.
In annotation workflows, miscalibration creates a specific problem: your review queue threshold, the confidence score below which frames get flagged for human review, is based on a number that doesn't mean what you think it does. If you're routing everything below 0.75 to review, but your model is actually only 55% accurate on frames it scores at 0.75, you're accepting a lot of bad labels without knowing it.
Building a calibration curve for your annotation model
The procedure is straightforward. Take a random sample of your pre-labeled outputs, perhaps 500 frames, and have annotators review and correct them without seeing the confidence scores. Then, for each corrected frame, record the original confidence score and whether the pre-label was accepted without modification.
Group your samples into confidence deciles: 0.5 to 0.6, 0.6 to 0.7, and so on. For each group, compute the acceptance rate: what fraction of pre-labels in that bucket were accepted by human reviewers without changes. Plot this against the midpoint of each confidence bucket. A well-calibrated model will show a line close to the diagonal. An overconfident model will show a line below the diagonal, especially at higher confidence scores.
Using calibration to set smarter thresholds
Once you have the curve, you can read off your actual accuracy at each confidence threshold instead of assuming it. If your calibration curve shows that 0.8 confidence corresponds to 72% acceptance rate, you can make a real cost-benefit decision: is 72% accurate enough to skip review on that fraction of frames, given your dataset quality requirements?
For most production annotation workflows, you want a threshold that corresponds to roughly 92 to 95% acceptance rate in the calibration curve, not 0.8 confidence score. The exact threshold depends on your domain. Medical imaging requires higher acceptance rates than retail object detection, where minor box imprecision has less downstream impact.
Class-level versus global calibration
Global calibration curves mask class-level variance that matters significantly for annotation. A model may be well-calibrated across all classes on average while being substantially overconfident on rare or visually complex classes. If pedestrians are your hardest category and your model is overconfident on pedestrian detections specifically, a global threshold misses the class you most need human review on.
Build class-specific calibration curves for any category where your model's training distribution is sparse or where annotation precision requirements are higher. Route those categories to review at a lower confidence threshold than your global setting, and monitor the per-class acceptance rates monthly to catch drift in specific categories before it affects your dataset quality.
Calibration techniques: temperature scaling and Platt scaling
When your calibration curve reveals significant overconfidence, post-hoc calibration techniques adjust the model's outputs without retraining. Temperature scaling is the most widely used approach for neural network classifiers: it divides the model's logits by a learned temperature parameter, which spreads the probability distribution and reduces overconfidence. The temperature parameter is learned on a held-out calibration set in a few minutes and does not require any changes to the model architecture or training procedure.
Platt scaling is an alternative that fits a logistic regression model to the pre-trained model's raw outputs, mapping them to calibrated probabilities. It provides more flexibility than temperature scaling when the miscalibration is non-monotone, such as when the model is overconfident in a specific score range but well-calibrated elsewhere. For annotation workflows, either technique is appropriate as long as the calibration set is representative of the production annotation domain.
Practical integration in annotation pipelines
The output of a calibration exercise is a lookup table or a simple function that maps raw model confidence scores to calibrated confidence scores. Integrating this into an annotation pipeline means applying the calibration function to the raw confidence scores before they're used to prioritize the review queue. The annotation tool displays calibrated scores, and reviewers are sorted by calibrated uncertainty rather than raw model output.
Teams that implement calibration typically adjust their review thresholds downward after calibration, because they discover that their previous threshold, set based on raw scores, was accepting more errors than intended. This is not a failure of the previous system; it's the expected finding when measuring something that was previously assumed rather than measured.
When calibration alone is insufficient
Calibration adjusts the mapping between confidence and accuracy but does not improve the underlying accuracy itself. A model that's well-calibrated but only 65% accurate on a class still requires human review on 35% of its predictions. Calibration makes the review prioritization correct; it doesn't reduce the amount of review needed for genuinely difficult classes.
For classes where even a well-calibrated model consistently falls below the accuracy threshold needed for pre-labeling to be time-positive, the investment is better placed in targeted fine-tuning on that class rather than better calibration. Fine-tuning with 200 to 500 high-quality labeled examples often produces a 10 to 15 percentage point accuracy improvement on the target class, which then makes calibrated confidence useful for that class's review queue prioritization.
Start calibrating your pre-label thresholds
