Label Noise: The Accuracy Ceiling Nobody Put There Deliberately

What problem does this solve?

A contract electronics manufacturer inspects reflow-soldered boards for defects: insufficient solder, bridging, tombstoning, cold joints and voids. Three inspectors label training images across two shifts. The vision model plateaus at 87.4% accuracy and stays there across three architectures and four training regimes.

The team eventually re-labels a sample of 500 images with two inspectors working independently. They disagree on 61 of them — just over 12%. Nearly all the disagreement is concentrated in one place: marginal joints where the fillet is acceptable to one inspector and insufficient to another.

That single measurement reframes the whole project. The model was not underperforming; it was reproducing a boundary that the inspectors themselves do not draw consistently. Twelve percent disagreement caps agreement with any single inspector at roughly 88%, which is almost exactly where the model sat.

How the solution works

Measure agreement before doing anything else. Have two inspectors label the same sample independently and compute Cohen's kappa per defect class. Raw percentage agreement flatters the result because it includes cases where both inspectors trivially agreed on a clean joint.

Classify the noise. Random noise is scattered evenly and behaves like a fixed handicap. Class-dependent noise is systematic — cold joints confused with insufficient solder in one direction more than the other. Instance-dependent noise concentrates on genuinely ambiguous images and is the hardest to remove.

Adjudicate rather than average. Where two inspectors disagree, a third senior reviewer decides, and the decision rule is written into the inspection standard. Averaging two opinions produces a label nobody would defend.

Build a gold-standard subset. A few hundred images adjudicated by consensus become the fixed evaluation set. Model performance is measured against that, not against whichever inspector happened to label the test batch.

1. 1
   Ambiguity enters the taxonomy A defect class has a boundary that the written standard does not pin down — how much fillet counts as insufficient.
2. 2
   Inspectors resolve it differently Each applies a personal threshold, consistent within themselves but not with each other.
3. 3
   The training set encodes both Two near-identical images receive opposite labels, so the loss surface contains a contradiction that no parameter setting resolves.
4. 4
   The model fits the average It converges on a boundary somewhere between the inspectors, disagreeing with each of them on part of the ambiguous region.
5. 5
   Evaluation inherits the noise The test set carries the same inconsistency, so the measured ceiling reflects the labelling process rather than model capability.

Reference architecture

Label quality is produced by four layers, and only the last is technical.

Deployment options: Board images may reveal customer designs and are frequently covered by manufacturing NDAs, so adjudication and gold-set construction happen inside the plant's approved environment rather than on a shared annotation service.

Key capabilities

Integrations

Label quality is governed in the inspection process and the annotation platform; the model pipeline only consumes the result.

Industry use cases

Implementation approach

1. 1
   Double-label a sample Have two inspectors label several hundred images independently, without seeing each other's decisions.
2. 2
   Compute kappa per class Report chance-corrected agreement per defect class, not overall percentage agreement.
3. 3
   Identify the noise type Inspect the confusion matrix for direction. Systematic asymmetry points at the written standard, not the inspectors.
4. 4
   Revise the standard and adjudicate Fix the criterion for the ambiguous case, then re-adjudicate the affected images against it.
5. 5
   Freeze a gold set Build a consensus-labelled evaluation set and version it, so future scores are comparable.

Security & deployment

Adjudication logs record which inspector labelled what and where they were overruled, which is employee performance data as well as quality data. Agree in advance how it will be used, keep it inside the plant's approved environment, and separate the quality purpose from any personnel process. Board imagery is customer intellectual property in most contract-manufacturing agreements and should not be sent to an external annotation service without explicit written permission.

A worked example

The 500-image adjudication sample, broken down by where the disagreement actually sits.

1. Raw agreement. 439 of 500 images matched, giving 87.8% raw agreement — which sounds acceptable until the class balance is considered.
2. Chance-corrected. 82% of images are clean joints that both inspectors label trivially. Cohen's kappa comes out near 0.61, which is moderate, not strong.
3. Where the disagreement sits. Of 61 disagreements, 47 are insufficient-solder versus acceptable. The remaining 14 are spread across the other four classes.
4. Direction of the error. Inspector A calls marginal joints acceptable roughly twice as often as inspector B. The noise is class-dependent and directional, not random.
5. After adjudication. A senior reviewer settles the 61 cases against a revised written criterion, and the criterion is added to the inspection standard.

The model was never the constraint. Kappa of 0.61 on the critical class explains an 87.4% plateau precisely. Fixing the written standard and re-adjudicating the affected images raises the achievable ceiling; changing the backbone does not.

In code

Two diagnostics do most of the work: measure agreement between annotators, then use cross-validated out-of-fold predictions to surface likely mislabels.

import numpy as np
from sklearn.metrics import cohen_kappa_score, confusion_matrix
from sklearn.model_selection import cross_val_predict, StratifiedKFold
from sklearn.linear_model import LogisticRegression

# 1. Inter-rater agreement on the doubly-labelled sample.
print("raw agreement:", round((rater_a == rater_b).mean(), 4))
print("cohen kappa:  ", round(cohen_kappa_score(rater_a, rater_b), 4))
for cls in np.unique(rater_a):
    mask = (rater_a == cls) | (rater_b == cls)
    print(f"  kappa[{cls}]:", round(
        cohen_kappa_score(rater_a[mask] == cls, rater_b[mask] == cls), 3))

# Direction of disagreement: is it symmetric or systematic?
print(confusion_matrix(rater_a, rater_b))

# 2. Confident-learning style screen using OUT-OF-FOLD probabilities.
# In-fold probabilities would be contaminated by the very labels under test.
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=3)
proba = cross_val_predict(
    LogisticRegression(max_iter=2000), X, y_noisy, cv=cv, method="predict_proba",
)
classes = np.unique(y_noisy)
self_conf = proba[np.arange(len(y_noisy)), np.searchsorted(classes, y_noisy)]

# Class-conditional thresholds, not one global cutoff.
thresholds = {c: proba[y_noisy == c, i].mean() for i, c in enumerate(classes)}
suspect = np.array([self_conf[i] < thresholds[y_noisy[i]] for i in range(len(y_noisy))])

print("flagged for review:", int(suspect.sum()), "of", len(y_noisy))
print("by class:", {int(c): int(((y_noisy == c) & suspect).sum()) for c in classes})

# IMPORTANT: send these to a human adjudicator.
# Auto-relabelling to the model's prediction teaches the model its own bias.
review_queue = np.argsort(self_conf)[:200]

Per-class kappa, a confusion matrix showing whether disagreement is directional, and a ranked review queue. The queue is an input to human adjudication, not a relabelling instruction — flipping labels to match model predictions removes exactly the examples that would have corrected the model.

Diagnostic checks

• Compute Cohen's kappa per class on a doubly-labelled sample. Below about 0.7 on the critical class, labels are the binding constraint.
• Compare raw agreement against kappa. A large gap means the raw figure is inflated by an easy majority class.
• Read the annotator confusion matrix for asymmetry. Directional disagreement indicates a standard problem rather than carelessness.
• Plot label distribution by inspector and by shift. A step change at a shift boundary is systematic noise.
• Rank training examples by out-of-fold self-confidence and inspect the lowest. Genuine mislabels cluster there.
• Check whether model errors concentrate on the same images the inspectors disagreed about; if so the model has hit the label ceiling, not a capability limit.

Limitations & prerequisites

• Confident learning surfaces candidates, not verdicts; every flagged item still needs human adjudication.
• Relabelling to match model predictions reinforces whatever bias the model already has and can make the ceiling worse.
• Instance-dependent noise on genuinely ambiguous cases cannot be fully removed, only made consistent by an explicit rule.
• Kappa depends on class prevalence, so values are not directly comparable across datasets with different defect rates.

The three noise types and what each requires

They present as the same plateau but respond to completely different interventions.

More training data helps only the first row. The other three require a change to how labels are produced, which is a process decision rather than a modelling one.

Key takeaways

• Inter-rater disagreement sets the accuracy ceiling; no architecture change moves it.
• Measure chance-corrected agreement per class, because raw agreement is inflated by the easy majority.
• Distinguish random, class-dependent and instance-dependent noise — only the first is helped by more data.
• Use model confidence to rank labels for human review, never to overwrite them automatically.
• A consensus-adjudicated gold set is what makes scores comparable between model versions.

FAQ

Related topics & solutions

• ADASYN: Generating Where the Model Is Actually Struggling
• Class Imbalance: When 95% Accuracy Means the Model Found Nothing
• Data Leakage: The Model That Only Works Before Deployment
• Smart Inspection
• Duplicate Data: When One Patient Becomes Four Records
• Random Undersampling: Throwing Away Data on Purpose
• SMOTE: Inventing Minority Examples Without Copying Them
• Outliers: Telling a Broken Sensor From a Broken Chiller