Annotation budgets are finite. The question of which frames to label next is therefore a real allocation decision, and the naive answer, label the next batch in capture order, is rarely the best one. Active learning is the practice of making that allocation decision based on what the model says it needs rather than what order frames arrived in.
The core intuition is that not all unlabeled frames contribute equally to model improvement. A frame where your current model is already 97% confident teaches it nothing new. A frame where the model is genuinely uncertain, where it can't decide between two plausible interpretations, is worth many confident frames in terms of the information it adds when labeled and added to training.
The three main acquisition strategies
Uncertainty sampling is the most straightforward strategy. Your model scores all unlabeled frames, and you queue the ones with lowest maximum confidence for human annotation. This works well when your model's confidence is well-calibrated. If it's overconfident on categories with training data gaps, uncertainty sampling misses exactly the frames it should be finding.
Margin sampling is a refinement that looks at the gap between the model's top two confidence scores. A frame where the model is 0.6 confident in class A and 0.58 confident in class B is genuinely ambiguous. A frame where it's 0.92 for class A and 0.04 for class B is not, even if the absolute score on the top class is the same. Margin sampling surfaces the truly ambiguous cases rather than just the low-confidence ones.
Core-set selection approaches the problem differently. Instead of querying by model uncertainty, they look at the embedding space of your unlabeled data and select frames that are maximally diverse relative to your already-labeled set. The goal is coverage of the feature space rather than uncertainty reduction. This is particularly useful when you have distribution shifts in your unlabeled pool that your current model doesn't know about because it hasn't seen those regions of the feature space.
Practical considerations for CV annotation
In computer vision annotation workflows, pure uncertainty sampling has a well-known failure mode: it over-selects frames from the hardest classes, which creates a training set that's unbalanced toward the difficult distribution and underperforms on the common distribution. The practical fix is to run uncertainty sampling within class buckets: take the top-N uncertain frames per class, then combine them. This maintains class balance while still concentrating labeling on the uncertain regions of each class's distribution.
For object detection specifically, frame-level uncertainty is sometimes a poor proxy for label value. A frame that contains 20 well-understood objects and one unusual one will have moderate confidence at the frame level because the 20 easy objects are all high-confidence. Object-level uncertainty sampling, looking at the confidence on individual predicted boxes rather than the frame aggregate, better identifies which frames contain the hard cases.
Combining active learning with pre-labeling
Active learning and pre-labeling are complementary rather than competing strategies. Active learning tells you which frames to label next. Pre-labeling tells you how to label them faster. The combination is: run active learning to identify your highest-information unlabeled frames, then route those frames through the pre-labeler to generate starting boxes, then have annotators correct rather than draw from scratch. You get both the budget efficiency of active learning sample selection and the time efficiency of correction-first annotation.
Stopping criteria and label budget allocation
A practical challenge with active learning in annotation workflows is deciding when to stop sampling. The theoretical answer is "when model uncertainty stops decreasing with additional labels," but measuring that in practice requires continuous retraining between batches, which is expensive. A more tractable stopping criterion for annotation teams is a performance plateau on your validation set: when three consecutive annotation batches of 500 frames each produce less than 0.5% mAP improvement, you've reached the point of diminishing returns for active sampling in the current data distribution.
At that point, the decision shifts to whether you need to acquire new data from a different distribution rather than labeling more from the current pool. Active learning is not a substitute for distribution coverage: it can't tell you that your capture setup is missing night-time conditions or that your deployed domain has a class your training set doesn't include. Distribution coverage analysis through clustering or embedding visualization is a separate tool that complements active learning rather than competing with it.
Implementation considerations for production CV teams
The infrastructure overhead of active learning is the main reason teams under-adopt it. Running inference over a pool of 50,000 unlabeled frames to score uncertainty requires either a running model server or the ability to spin one up on demand. For smaller teams without dedicated MLOps infrastructure, this is a meaningful barrier.
A lighter-weight approach that captures most of the benefit is random-within-cluster sampling. Cluster your unlabeled pool by visual similarity using image embeddings from a pre-trained model, then sample randomly from each cluster while ensuring coverage across all clusters. This doesn't require active uncertainty estimation and can be implemented with any embedding model, but it avoids the extreme oversampling of common cases that purely random sampling produces.
Pre-labeling fits into this workflow at the batch level. Once your active learning system or clustering approach identifies the next batch of frames to label, those frames enter the pre-labeling queue before reaching human annotators. The annotation team sees a review task, not a drawing task, regardless of whether the frames were selected by uncertainty sampling or random-within-cluster.
Tracking label efficiency over time
The right metric for evaluating active learning in annotation is label efficiency: mAP improvement per labeled frame, tracked across batches. If your first batch of 1,000 frames produced 0.12 mAP improvement per frame and your fifth batch is producing 0.04 mAP improvement per frame, you can see the diminishing returns curve directly. This gives annotation managers a concrete basis for budget decisions: at current label efficiency, how many more frames do we need to reach the target performance threshold?
Start pre-labeling your active learning samples
