Most CV teams track experiments in MLflow but version datasets informally: a folder name with a date, a README with a sentence about what changed, or nothing at all. When a model underperforms and you want to trace the cause, an unlabeled dataset is a black box. You know the model weights and the hyperparameters but not which labels it trained on.
Dataset versioning in MLflow is a solved problem technically, but it requires a deliberate connection between your annotation pipeline and your experiment tracking. This article walks through how to wire that connection so every MLflow run records the exact dataset version it trained on, and you can reproduce any experiment from a checkpoint and its corresponding dataset hash.
The core concept: datasets as MLflow artifacts
MLflow tracks metrics, parameters, and artifacts per run. A dataset export is an artifact. When you trigger a training run, the first step is to export the current state of your labeled dataset as a COCO or YOLO-format archive, log it as an artifact for the current run, and record its SHA-256 hash as a run parameter. Any future run that trains on the same dataset will have the same hash, making it trivial to group runs by dataset version.
The hash-as-parameter approach is lightweight: you don't need MLflow's full Dataset API or a dedicated feature store. A 10-line Python function that computes SHA-256 over the sorted annotation file paths and their contents gives you a stable identifier that changes precisely when the label content changes.
Wiring the annotation export trigger
The natural integration point is your annotation platform's export webhook or scheduled export. Configure the export to write a COCO JSON to a shared storage location your training script can read. When the training script starts, it computes the dataset hash, logs it to the current MLflow run, then proceeds with training.
A more explicit flow uses a pipeline script that orchestrates both steps: export dataset, verify hash, start MLflow run with hash as parameter, trigger training. The advantage of the explicit script is that the dataset export and the run start are atomic from the pipeline's perspective, so there's no window where training starts before the export completes.
Tracking annotation quality metrics alongside model metrics
Once your dataset version is recorded per run, it's natural to also log annotation quality metrics: how many frames in this dataset version were corrected by a human versus accepted from pre-labels, the average confidence of accepted pre-labels, the inter-annotator agreement on the double-annotated sample. These become leading indicators for model performance. If a run trains on a dataset with low IAA scores, you can hypothesize label noise as a contributing factor without having to re-examine individual annotations.
Log these as MLflow parameters at the start of the run (they're properties of the dataset, not the training process) so they're available for filtering and grouping in the MLflow UI without needing to inspect individual artifacts.
Reproducing a historical run
When you need to reproduce a run from six months ago, the procedure is: find the run in MLflow, read the dataset_hash parameter, search your artifact store for the export with that hash, restore it to the training environment, and run training with the same hyperparameters. Because the hash is computed from annotation content rather than timestamps or filenames, the reproduced dataset is guaranteed to have the same labels as the original, even if the filenames or storage locations have changed.
This reproducibility guarantee is the main reason to invest in the integration. Without it, you can reproduce the model weights but not the dataset that produced them, which limits your ability to investigate model behavior at a specific point in the model's history.
Automating the export-to-training handoff
The manual step that kills annotation-to-training velocity is the export handoff: someone remembers to export the latest labeled data, drops it in a shared folder, and triggers the training job. Automating this step with a simple webhook reduces the delay between annotation completion and training kickoff from days to hours.
The webhook pattern is straightforward. Configure your annotation platform to call a training trigger endpoint when a batch is marked as complete. The trigger script exports the dataset with the current labels, computes the hash, starts the MLflow run, and kicks off the training job. The annotation manager marks a batch done, and training starts within minutes without a manual handoff.
Handling label versioning within a batch
Large batches often span multiple annotation sessions with corrections happening over days. If a training run starts mid-batch, some frames will have been reviewed and some won't. The clean solution is to train only on batches that have been marked as complete and QA-reviewed, treating the in-progress batch as a staging area that doesn't enter training data until it's done.
The practical risk is that the complete batch model is trained less frequently. A reasonable compromise is to allow partial batches of at least 500 reviewed frames to trigger a training run, with the hash computed over only the finalized portion. When the full batch completes, a second training run uses the complete batch hash, and the model history shows both runs with their respective dataset versions.
Querying annotation history in MLflow
Once dataset hashes are tracked as MLflow run parameters, they become queryable. You can ask: which runs were trained on dataset versions with inter-annotator agreement above 0.8? Which runs used a pre-labeler version prior to the October calibration update? MLflow's built-in search syntax supports parameter filtering, so these queries run without additional tooling.
Building a dataset quality dashboard from MLflow query results is a useful artifact for annotation review meetings. If model performance dropped between two runs and both runs show similar hyperparameters, the dataset parameter differences are the first place to investigate. The annotation quality metrics logged with each run make that investigation fast rather than requiring manual review of labeling records.
Export versioned datasets from Annotgrove
