Data labeling when experts are expensive

The scarce resource is not labels, it is attention

In an ordinary machine learning program the model is the expensive part and the labels are the cheap part. When ground truth requires a licensed professional, that ratio flips and stays flipped. A pathologist, a structural engineer who will put a seal on the drawing, a certified health physicist, an attorney reading contract language, a claims adjuster with signature authority: each has a full calendar, a queue of billable or clinical work, and no organizational incentive to spend an afternoon in a labeling tool. The supply of judgment is fixed for the life of the project. Every design decision after that follows from one question, asked before the first case is queued: what is this hour for?

Teams that skip the question do a predictable thing. They buy an annotation platform, load ten thousand records, hand the expert a login, and discover four weeks later that the expert labeled two hundred items, most of them easy, and stopped answering email. The project then has a small pile of labels of unknown reliability, no measure of how much the expert disagrees with a second expert, and no evaluation set anyone would defend. The money was spent. Nothing was bought.

The alternative is to treat expert time the way a program office treats a limited flight-test window. You do not fly to find out whether the aircraft is interesting. You fly a plan, in a fixed order, with the highest-information sorties first, and you instrument everything. Expert labeling is the same problem with a different unit of scarcity.

Separate judgment from transcription

The first economy is almost always available, and almost always missed. Look at the labeling task and split it into two piles: the part that requires the license, and the part that merely requires care.

A radiologist marking whether a nodule is present is exercising judgment. The same radiologist typing the study accession number into a form is doing data entry at a specialist's rate. An attorney deciding whether an indemnity clause is mutual is exercising judgment. The same attorney scrolling to find the indemnity section is doing retrieval. In practice the judgment fraction of a naive labeling workflow is often well under half the wall-clock time, and the rest is navigation, context loading, tool friction, and typing.

So build the workflow backward from the judgment. Pre-populate every field the expert does not need to decide. Pre-locate the region, the clause, the exposure record, the segment of the waveform. Put the decision on one screen with a keyboard shortcut per label value and a required one-line rationale field. Then measure seconds per decision, not items per day, because seconds per decision is the number that tells you whether the workflow or the expert is the bottleneck.

Build the evaluation set before the training set

Most teams spend their first expert hours generating training data. That is the wrong order when expert hours are capped. Training labels can be supplemented, corrected, weakly supervised, or partly replaced by other signal. Evaluation labels cannot. The evaluation set is the instrument you will use to decide whether the system works, whether it is safe to deploy, and whether the vendor gets paid. An instrument built from cheap labels measures nothing.

So the first tranche of expert time buys a small, adjudicated, frozen evaluation set, stratified so the rare and consequential classes are represented well enough to estimate a rate on. That set is versioned, sealed, and never used for training, tuning, threshold selection, or prompt iteration. If it leaks into development, it stops being an instrument and becomes a scoreboard the team has learned to game. We write this rule into acceptance language on delivery work; the mechanics are covered in acceptance criteria for a machine learning deliverable.

How many expert-labeled items you actually need

The number is smaller than people fear and larger than people hope, and it is arithmetic, not opinion. If you want to estimate a rate — sensitivity on a class, false-extraction rate, agreement with the reference standard — the width of the confidence interval around that estimate is set by the sample size. For a binomial proportion at 95% confidence, worst case, the required count is about 0.96 divided by the square of the half-width you will accept.

Precision on one rate	Adjudicated items needed	What that number is good for
±10 points	97	A smoke test. Enough to see a catastrophe, not enough to defend a threshold.
±5 points	385	The usual working target for a single class in a pilot evaluation.
±3 points	1,068	Acceptance testing where a few points of performance change the decision.
±2 points	2,401	Regulatory-grade claims and head-to-head comparisons against a predicate.
Zero errors observed	3 ÷ n upper bound	300 clean items supports "under 1%" at 95% confidence — the rule of three (Hanley and Lippman-Hand, JAMA, 1983).

Two things make the real number smaller than the table suggests. First, these counts are worst case, computed at a rate of 50%; for a rate near 5% or 95% the same precision needs far fewer items. Second, you rarely need every class measured to the same precision. Decide which one or two rates carry the deployment decision, size those properly, and accept wider intervals on the rest. One thing makes the number larger: if the same items must support several stratified estimates, the strata each need their own count.

This arithmetic is worth showing to the expert before asking for a single label. "We need four hundred adjudicated cases, here is why, here is what it will support, here is what happens if we stop at one hundred" is a conversation a professional can plan around. "Please label as much as you can" is not.

Experts disagree, and the disagreement is the most valuable thing you will measure

The single most common failure in expert-labeled programs is treating one expert's opinion as truth. Published data on professional agreement should end that habit. In the B-Path study (Elmore and colleagues, JAMA, 2015), 115 practicing pathologists interpreted breast biopsy slides and were compared against an expert consensus reference. Overall concordance across 6,900 interpretations was 75.3%. It was not uniform: 96% for invasive carcinoma, 87% for benign without atypia, 84% for ductal carcinoma in situ, and 48% for atypia. Same specialty, same slides, same year. The class where the diagnosis changes patient management is the class where the professionals split nearly evenly.

That is not an indictment of pathology. It is a description of what a hard judgment task looks like from the inside, and every domain with a licensure requirement has classes that behave like atypia. The engineering consequence is direct: before you can say a model is right or wrong, you need to know how often two qualified humans say different things about the same item, class by class.

Report that with a confusion matrix first and a coefficient second. Chance-corrected agreement statistics are useful and easy to misread. Feinstein and Cicchetti documented in 1990 that a pair of raters can agree on nearly every item and still produce a low kappa when one category dominates the sample, and that the same observed agreement can yield different kappas depending on how the marginal totals are unbalanced. A single number without the matrix behind it hides whether you have a guideline problem in one label pair or noise spread everywhere. For Krippendorff's alpha, the conventional reading is that values at or above 0.800 support firm conclusions and values between roughly 0.667 and 0.800 support tentative ones only.

Your model cannot be more right than your reference standard

Label error is not a hypothetical. Northcutt, Athalye and Mueller found label errors throughout the test sets that the field grades itself on: an average of 3.4% across ten widely used benchmarks, with roughly 6% of the ImageNet validation set mislabeled, verified by human review of algorithmically flagged candidates. Those are the datasets with the most eyes on them in the history of the discipline. A first-pass, single-reader, deadline-pressured expert label set in a specialist domain will not do better on its own.

Practically, this means the reference standard is a design artifact with its own specification, not a byproduct. Write down who reads, whether reads are blind to each other, who adjudicates, whether the adjudicator sees the prior reads, and what evidence the adjudicator is allowed to consult that the original readers were not. A panel that adjudicates with follow-up outcome data available produces a stronger reference standard than one working from the same inputs, and a materially different one. Both are legitimate. Confusing them is not.

Five ways to spend an expert hour

There is no single correct labeling protocol. There is a menu, and the choice is a budget decision with measurable consequences.

Protocol	What the expert does	Where it breaks
Blind double read plus adjudication	Two experts label independently; a third resolves conflicts	Highest cost per item. Also the only protocol that measures the reliability of your own reference standard, which is why the evaluation set should use it.
Single read with audit sample	One expert labels; a random fraction is re-read blind	Cheap and defensible if the audit fraction is sized to detect the error rate you care about. An unsized 5% audit detects almost nothing.
Assisted review	A model proposes; the expert accepts or corrects	Large time savings, and a measured pull toward whatever the model said. Needs a control arm to quantify.
Tiered triage	A model or trained non-expert clears routine cases; the expert sees only flagged ones	Only valid if the triage step's miss rate has been measured on expert-labeled data first. Otherwise the expert never sees the cases that matter.
Programmatic weak supervision	The expert writes rules and heuristics instead of per-item labels	Scales to unlabeled data, but still needs a small expert-labeled set to validate against, and rules encode blind spots at scale.
Language-model pre-labeling	A model drafts labels; the expert validates a stratified sample	Strong where the task is well specified in text. Unproven as a stand-in for licensed judgment on the classes where licensed professionals disagree with each other.

Assisted labeling works, and it charges an anchoring tax

The time savings from model-assisted annotation are real and large. In the MedSAM work published in Nature Communications in 2024, an assisted pipeline — expert draws a sparse marker, the model segments, the expert corrects — cut annotation time by 82.4% and 83.0% for the two experts in the study. On segmentation and localization tasks, this is the single biggest lever available.

The cost shows up in a different column. Dratsch and colleagues, writing in Radiology in 2023, gave 27 radiologists mammograms alongside suggested BI-RADS categories from an AI system, some of them deliberately wrong. When the suggestion was correct, readers were right about 80% of the time across experience levels. When the suggestion was wrong, correct assessments fell to 19.8% for inexperienced readers, 24.8% for moderately experienced readers, and 45.5% for very experienced readers. Experience helped. It did not protect.

The design implication is not to abandon assisted labeling. It is to stop pretending assisted labels and independent labels are the same product. Use assistance for volume on the training set. Keep the evaluation set independent, or at minimum keep a control arm — a fraction of evaluation items labeled with no model suggestion visible — so the anchoring effect is measured rather than assumed. And never let the model that generated the pre-labels be the model you are grading against that set.

Write down the rules instead of collecting the clicks

Weak supervision inverts the interview. Instead of asking the expert to label ten thousand items, you ask them to state the rules they would apply, encode each rule as a function over the raw data, and let a statistical model reconcile the rules where they conflict. This is the data programming approach behind Snorkel (Ratner and colleagues), which learns the accuracies and correlations of noisy labeling functions from their agreements and disagreements, without ground truth, and emits probabilistic labels over a much larger unlabeled corpus. The published system reports deployments including work with the Department of Veterans Affairs and the Food and Drug Administration.

For expert-scarce domains this is often the highest-value hour on the whole project, because it converts a professional's tacit heuristics into an asset that keeps producing after the meeting ends. Three practical notes. Rules are easier to elicit from a disagreement than from a blank page, so run the rule-writing session immediately after adjudication, using the cases that split. Rules must be versioned like code, because a changed rule invalidates every label derived from it. And weak supervision does not remove the need for an expert-labeled evaluation set; it removes the need for an expert-labeled training set. Those are different budgets.

Where language-model pre-labeling holds, and where it does not

The published evidence for language models as annotators is genuinely strong on general text tasks. Gilardi, Alizadeh and Kubli, writing in PNAS in 2023, reported that on four datasets of tweets and news articles, zero-shot ChatGPT accuracy exceeded that of crowd workers by roughly 25 percentage points on average, with better intercoder agreement, at under three tenths of a cent per annotation.

Read the comparison carefully before generalizing it. The baseline was crowd workers on relevance, stance, and framing tasks. It was not a board-certified specialist on a diagnostic call, a licensed engineer on a load path, or an attorney on an indemnity question. The honest position in 2026 is that model pre-labeling reliably beats untrained human annotation on well-specified text, that it has not been shown to substitute for licensed judgment on the hard minority of cases in regulated domains, and that the hard minority is exactly where your project's risk lives. Use it to clear the easy mass, measure how it does on a stratified expert-labeled sample, and route by measured confidence rather than by hope.

Active learning buys speed and can cost you a clean evaluation set

Pool-based active learning — train on a small labeled seed, score the unlabeled pool, send the model's most uncertain items to the expert, retrain, repeat — is the textbook answer to a fixed labeling budget, and it usually works. It concentrates expert attention on the decision boundary instead of on the thousand easy cases the model already handles.

Two cautions that get skipped. Uncertainty sampling produces a labeled set that is deliberately not representative of the deployment distribution, so it is training data, not evaluation data, and reusing it as evaluation data will make the system look worse than it is on the easy mass and better than it is on nothing at all. Second, the loop needs a stopping rule agreed in advance — a target interval width, a plateau in adjudicated corrections, or a fixed hour budget — or it will consume every hour the expert has and end when the expert quits rather than when the data is sufficient.

Design the session so the hour is worth what it cost

Batch by class, not by chronology. Twenty consecutive cases of the same decision type are far faster per item than twenty mixed ones, because the expert loads the criteria once.

Cap the session. Agreement degrades with fatigue in every domain that has studied it. Two focused hours produce better labels than a scheduled full day, and a professional will actually show up for two hours.

Capture the rationale, not just the label. One sentence per hard case. This is the raw material for the guideline revision, for the weak-supervision rules, and for the eventual explanation of why the reference standard says what it says.

Version-stamp every label. The label of record carries the guideline version it was produced under. Without it, a mid-project guideline change quietly turns your dataset into two incompatible datasets.

Show the expert the effect. Send back a short note after each round: here is what changed, here is where the model still fails, here is the case you resolved that moved the number. Domain professionals stay engaged when the work visibly matters and disengage when it feels like piecework.

The compliance layer changes who may touch the data

In regulated domains the labeling protocol is a compliance artifact before it is an engineering one. Three constraints come up on nearly every engagement.

Identifiers. Under the HIPAA Privacy Rule, 45 CFR 164.514(b) provides two routes to de-identified data: a formal Expert Determination that re-identification risk is very small, and the Safe Harbor removal of the 18 enumerated identifier types combined with no actual knowledge of residual risk. The complication in clinical labeling is that the reader often needs context — prior studies, dates, sequence — that Safe Harbor strips. That usually keeps the labeling environment inside the covered entity's boundary rather than outside it, which changes the platform decision, the network path, and the contract. Related patterns are covered in running a model where the data cannot leave.

Professional responsibility. The NCEES Model Law defines responsible charge as direct control and personal supervision of engineering work, and a licensee may seal only work performed by them or under their responsible charge and within their area of competence. "A licensed engineer reviewed the model's output" and "a licensed engineer is in responsible charge of the result" are different statements with different liability. Buyers should say which one they are contracting for, in writing, before the first label.

Moving regulatory expectations. Two anchors are worth naming precisely because they are in motion. FDA finalized its marketing-submission guidance on Predetermined Change Control Plans for AI-enabled device software functions in December 2024; the broader draft guidance on lifecycle management for AI-enabled device software functions, issued in January 2025, remains a draft and appears on CDRH's fiscal-year 2026 list for finalization. NIST published the AI Risk Management Framework 1.0 on January 26, 2023, with its GOVERN, MAP, MEASURE and MANAGE functions, and has said a revision is underway. Pin the specific version and date you are complying against in your documentation, and expect to restate it.

What to write into the labeling scope of work

• The reference standard, specified: who reads, whether reads are blind, who adjudicates, and what the adjudicator may see.
• A frozen, adjudicated evaluation set that no training, tuning, or prompt iteration ever touches.
• Named precision targets per class, expressed as interval width, with the item count that supports them.
• Agreement reported as a confusion matrix plus a coefficient, never a coefficient alone.
• Guideline version stamped on every label of record, with rework terms if the guideline changes mid-run.
• Whether model-assisted pre-labels are permitted, and the control arm that measures the anchoring effect.
• Expert hours costed as a separate line from platform and engineering, with a capped, scheduled draw.
• Data handling terms matched to sensitivity: enclave, retention, deletion, and who may see identifiers.

Bottom line

When ground truth needs a license, the project is a budgeting exercise wearing an engineering costume. Spend the first hours on the guideline and the evaluation set, not on volume. Measure how much two qualified professionals disagree before you measure the model, and expect the disagreement to be concentrated in the classes that matter most. Use assistance, weak supervision, and model pre-labeling to buy scale on the training side, and keep the measuring instrument clean. Teams that do this ship systems whose numbers survive review. Teams that hand an expert a login and hope generally do not, and they find out late.

Frequently asked questions