Teams rolling out AI-assisted data labeling almost always watch one number: how many labels get produced per hour. Once a project scales, that number stops meaning much. A model can repeat a pattern all day long, but it can’t tell you whether the pattern was ever agreed on in the first place. “Label counts measure motion, not readiness,” says Dhara Patadia, an AI specialist working with training data and annotation workflows. “The number that predicts whether automation will work is how often two trained annotators land on the same answer without talking to each other.” That reframes annotation quality as something you check before automating, not after.
Speed doesn’t mean the work is done
An AI labeling tool can spit out thousands of suggested labels an hour and still leave a team with more work than before. Volume was never the problem; near-correct results eat more review time than obviously wrong ones do. A label that’s almost right looks fine at a glance and takes real attention to catch. A blatant error gets fixed in a second. So a team can look busier and still not be saving any effort at all; the work just moved from labeling to checking.
A case in point: Multilingual intent labeling
This showed up on a project labeling customer messages for intent across several languages, where annotators had to assign a single intent category to messages that often carried more than one plausible reading. The written guidelines covered the obvious cases, but plenty of messages mixed a complaint with a question, or used idioms that didn’t map cleanly onto any category, and different annotators started resolving the same ambiguity differently. One person might label a sarcastic thank-you as a complaint. Another might take it at face value and mark it as satisfaction.
The disagreement wasn’t evenly spread across languages either. It clustered in languages where the team had fewer native-speaking annotators, so borderline calls got made by people working from translations instead of the original phrasing. Once an AI tool was brought in to generate first-pass labels, it inherited that inconsistency. It had been trained on data that was labeled under rules nobody had agreed on, so there was no stable pattern for it to learn. The mislabeled sarcasm, the flattened mixed-intent messages, those weren’t random model failures. They were a direct result of the humans upstream not agreeing on what counted as what, and the gap was worst exactly where oversight was thinnest.
“You can hand a model a thousand examples of disagreement, and it will learn the disagreement, not the correct answer,” Dhara says. “It has no way of knowing which annotator was right.”
Fix the disagreement, not the symptom
Correcting labels one at a time never solved anything. What worked was changing how the team handled cases the guidelines didn’t cover. Annotators were told to flag anything unfamiliar instead of guessing, and the old rule stood until a subject-matter expert, in this case someone fluent in the language in question, reviewed the example. Once a call was made, it went into the shared guideline and applied to every similar case going forward. That stopped the same disagreement from quietly spreading through the dataset, and it stopped the weakest-coverage languages from silently accumulating the most inconsistent labels.
It also changes what review is for. A reviewer isn’t just catching mistakes after the fact. They’re deciding where a category’s edges sit and whether a pattern is stable enough to hand off to automation. “A reviewer who only fixes labels is working one step too late,” Dhara says. “By the time a mistake reaches them, it’s already been made ten more times upstream.” That’s a design decision, not a quality check tacked on at the end.
How to tell if a task is ready
Tasks with clean boundaries, unambiguous single-intent messages, and categories nobody would argue about tend to get high agreement between trained annotators, and that’s where automation earns its keep. The tool hands the reviewer a starting point, and the reviewer just checks it instead of building it from scratch.
Subjective judgment is where this breaks down. Even something as simple as picking between two candidate responses can hinge on several quality criteria the team never wrote down. The interface looks simple, but the judgment behind it runs much deeper. Hand that kind of task to a model too early and it’ll still give you a confident answer; it just won’t tell you that the answer reflects whichever interpretation happened to dominate the training data, not some objectively correct one.
Guidelines don’t stay fixed, and that’s fine
There’s a reason to keep tracking agreement past the first week of a project: real data keeps producing cases nobody planned for. A new slang term shows up in one language and takes weeks to reach the others. A product change shifts what “complaint” even means for a given feature. Data prep turns into ongoing maintenance rather than a box you check before training starts, and someone needs to keep track of which guideline version each batch was labeled under.
“Guidelines are never finished, they’re just current,” Dhara says. “The moment a team treats them as a fixed spec, agreement starts drifting, and nobody notices until the dataset is already inconsistent.”
The role for AI here is to take on more of the clear-cut cases over time and flag the ones it’s unsure about. Deciding which interpretation is correct when the real world doesn’t offer a clean answer stays a human call, which is exactly why agreement between annotators, not the label count, is the number worth checking first.
Featured image credit
