Resources/Training Images, Labels, and Thresholds: How to Validate a Visual Inspection Model
Operational Excellence

Training Images, Labels, and Thresholds: How to Validate a Visual Inspection Model

A model outputs a score, and your plant picks the threshold. Here is how to collect and label inspection images, choose a threshold per defect, avoid inflated test results, and know when a model is ready to trust.

13 min read
By Monitory team

Move the threshold

One score. Two errors. Your call.

Illustrative replay
Good partsDefective partspassreject0255075100

At this threshold

Escapes
1
defective parts passed
False rejects
1
good parts rejected

Target for this defect

Zero escapes. Some false rejects are acceptable.

Target not met

Keep the line low. A reworked good part costs far less than a missing clip in the field.

Lower the line to catch more defects. Raise it to protect good parts.

The scores in this chart are synthetic and exist only to illustrate the tradeoff. Real score distributions, overlap, and targets come from your own validation set, split by part, lot, or date.

The longest discussion in our session with the machine vision engineering team was not about cameras or models. It was about a number most people never see: the threshold. A model does not say "reject." It outputs a score, and somebody decides which score means reject. That decision, and the test data behind it, decides whether operators trust the station or start overriding it.

This article is the fifth in our visual inspection series. The first article covered choosing rule-based checks, supervised AI, or anomaly detection for each defect. This one is the how-to for the step after that: collecting and labeling images, choosing a threshold per defect, and building a test set that tells you the truth before the model meets production.

Start with the images your line actually produces

The training set should look like your line on an ordinary day and on a bad one. That means images from every shift, supplier lot, product variant, and lighting condition the station will see, captured on the station itself rather than on a bench.

Two kinds of images matter most:

  • Defects, including borderline ones. The parts that are hard to call are the ones the model will be judged on.
  • Harmless variation. Coolant film, tool marks, color shifts between lots, and dust that wipes off. If the model never sees these labeled as good, it will reject them.

Defects are rare by nature. Researchers working on surface defect detection note that defective items can be a hundred, a thousand, or even a million times less likely than good ones [1]. You will not collect a balanced dataset, and you do not need to.

Spend detailed labels where they pay off

Labeling every defect with a precise outline is slow. Research on mixed supervision shows you can label most images simply as good or bad and outline only a few. On a real production dataset with 2,085 good and 246 defective training images, detection performance (average precision) rose from 73.3% with no outlines, to 83.2% with 16 outlined defects, to 95.4% with every defect outlined [1].

A second result from the same study is more surprising. On a steel strip dataset, ten times more defective images (300 to 3,000) improved average precision by only 3.8 points, from 90.3% to 94.1% [1]. Adding 50 detailed outlines to the original 300 images reached 94.2% [1]. More images helped less than a few careful outlines. Those results come from specific datasets, so treat them as direction rather than a forecast for your line.

Labels are measurements, so check them like measurements

Every label is someone's judgment, and judgments disagree. Even widely used machine learning test sets carry label errors: a study of ten common benchmarks found at least 3.3% of test labels wrong on average, and enough noise to change which model looked best [2]. Industrial labels are made by people under time pressure, on borderline parts, so assume yours have errors too.

The human baseline is also lower than most teams expect. In a Sandia National Laboratories study of 82 inspectors judging 140 precision parts, inspectors caught 85% of defective parts but also rejected 35% of acceptable ones [3]. No acceptable part was accepted by every inspector, and only 35% of correct rejects named the actual defect [3]. That test used a 30% defect rate in an isolated setting, and the author treats the results as an upper bound on real-world performance [3]. The same paper cites earlier work putting typical visual inspection accuracy at 70% to 80% [3].

Two practices follow from that evidence:

1. Write a labeling guide with limit samples. For each defect, keep physical or photographed examples of "just acceptable" and "just rejectable," and label against them. 2. Measure agreement before you trust labels. Attribute agreement analysis scores pass or fail judgments against a reference. The automotive industry guideline cited in statistical software documentation treats a kappa of at least 0.75 as good agreement [4]. If your labelers do not reach it, fix the guide before training on their labels.

A model can only be as consistent as its labels

If two experienced inspectors disagree on a part, the model will learn that disagreement. Resolve borderline cases into written limit samples first. A model trained on inconsistent labels tends to look unreliable, when the real cause is the labels.

Why accuracy is the wrong number

When defects are rare, accuracy hides failure. Google's machine learning guidance gives the example directly: at 1% prevalence, a model that always predicts the majority class scores about 99% accuracy while catching nothing [5]. An inspection model that never rejects anything would look excellent on paper.

Use two numbers instead, both stated in plant language:

  • Escape rate: the share of defective parts the model passes. In machine learning terms this is one minus recall.
  • False reject rate: the share of good parts the model rejects. This is the false positive rate.

Google's guidance also recommends precision and recall curves over ROC curves when classes are heavily imbalanced, because they show the tradeoff for the rare class more clearly [6]. The NIST AI Risk Management Framework asks for exactly these false positive and false negative rates, broken out by data segment [7]. For a station, the segments are defect type, product variant, shift, and supplier lot.

Choose the threshold per defect

A score threshold of 0.5 is a default, not a decision. Google's guidance notes that 0.5 is a poor choice when the two kinds of error have different costs [8]. On an inspection line they almost always do.

Use the interactive chart at the top of this article to move a threshold across synthetic good and defective scores and watch both counts change. Then set each defect's threshold from its own cost:

DefectCost of an escapeCost of a false rejectThreshold directionAcceptance test
Missing safety clipField failure, safety riskOne good part reworkedLow: reject anything doubtfulZero escapes on the full defective test set
Scratch in sealing zoneLeak claim from customerScrap of a machined partLow to middleZero escapes above the size limit
Blemish on a hidden faceMinor cosmetic complaintScrap or rework of a good partHigher: protect yieldFalse rejects below your yield target
Unknown damage (anomaly flag)Depends on what it isReviewer timeRoute to review, not rejectReview queue fits reviewer capacity

Write the target for each row before you tune anything, then move the threshold until the test set meets the target. If no threshold meets both targets for a defect, the answer is better images, lighting, or labels, not a compromise threshold.

Build a test set that cannot fool you

The most common way to get a misleading result is leakage: information from the test set reaching training. A review of machine learning studies found leakage problems in 329 papers across 17 fields and described eight distinct types [9]. Three of them show up in inspection constantly:

  • Duplicates. Several images of the same part, or frames from the same video, split between training and testing.
  • Non-independent samples. Parts from the same lot or fixture in both sets, so the model learns the lot rather than the defect.
  • Time. Training on later production and testing on earlier, which hides how the model handles change.

Split by part, lot, shift, or date, never by individual image. Then keep the result honest over time:

1. Freeze a regression set. Keep a fixed set of labeled images, including every borderline case and every past escape, and never train on it. 2. Rerun it after every change. New model, new threshold, new light, new lens: rerun the regression set and compare escape and false reject rates per defect against the previous version. 3. Add every production escape. When a defect gets through, add its image to the regression set so it can never quietly return.

NIST's framework calls for test sets representative of conditions of expected use, documented test methods, and monitoring in production [7]. A frozen regression set rerun after every change is the plant-floor version of that guidance.

Where Monitory fits

Threshold and model changes are maintenance events for the station. Monitory records them against the inspection asset alongside its regression results, so when escape or false reject rates move, the team can see what changed and when. If you have a model heading toward production, request the free validation plan review at the end of this article. The same record links inspection trends to the upstream equipment that produced the parts. See computer vision on the Monitory platform.

Summary

A model is ready when it meets a written escape and false reject target for each defect, on a test set that reflects production and shares no parts, lots, or time periods with training. Accuracy, benchmark scores, and default thresholds do not tell you that.

Next steps for one station:

1. Collect images on the station across shifts, lots, and variants, including harmless variation. 2. Write a labeling guide with limit samples and check labeler agreement. 3. Label most images good or bad, and outline a small, chosen subset of defects. 4. Write an escape and false reject target per defect, then set each threshold to meet it. 5. Split test data by part, lot, or date, freeze a regression set, and rerun it after every change.

For choosing the method in the first place, read how to choose rule-based, supervised, or unsupervised inspection. For the path from a detection to a disposition, see computer vision quality inspection: from defect detection to corrective action.

Frequently asked questions

How many images do we need to train an inspection model?

There is no independent, general minimum. It depends on how varied the defects and the good parts are. Research shows that a few detailed defect outlines can matter more than many additional images [1], so start small, measure escape and false reject rates on a proper test set, and add images where the errors are.

Should the model be judged against a perfect standard?

Judge it against your written targets and against your current inspection. Trained human inspectors in a controlled study caught 85% of defective parts while rejecting 35% of good ones [3]. A model that meets your per-defect targets on a leak-free test set is a reasonable bar, not perfection.

What is a good threshold for an inspection model?

The one that meets your escape and false reject targets for that defect. A single threshold of 0.5 for every defect ignores that escapes and false rejects cost different amounts [8].

How do we know if our labels are good enough?

Have several labelers judge the same set of parts against limit samples and run an attribute agreement analysis. A kappa of at least 0.75 is the commonly cited guideline for good agreement [4]. If agreement is lower, improve the labeling guide before training.

Why did our model test well and then fail on the line?

Usually because the test set did not reflect production, or because of leakage, such as images of the same part or lot in both training and testing [9]. Rebuild the test set with splits by part, lot, or date, and include conditions from every shift and supplier.

References

[1] J. Božič, D. Tabernik, D. Skočaj, "Mixed Supervision for Surface-Defect Detection: From Weakly to Fully Supervised Learning", Computers in Industry, 2021. https://arxiv.org/abs/2104.06064

[2] C. G. Northcutt, A. Athalye, J. Mueller, "Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks", 2021. https://arxiv.org/abs/2103.14749

[3] J. E. See, "Visual Inspection Reliability for Precision Manufactured Parts", Human Factors, 2015. https://www.osti.gov/pages/biblio/1236203-visual-inspection-reliability-precision-manufactured-parts

[4] Minitab Support, "Key results for Attribute Agreement Analysis". https://support.minitab.com/en-us/minitab/help-and-how-to/quality-and-process-improvement/measurement-system-analysis/how-to/attribute-agreement-analysis/attribute-agreement-analysis/interpret-the-results/key-results/

[5] Google, "Classification: Accuracy, recall, precision, and related metrics", Machine Learning Crash Course. https://developers.google.com/machine-learning/crash-course/classification/accuracy-precision-recall

[6] Google, "Classification: ROC and AUC", Machine Learning Crash Course. https://developers.google.com/machine-learning/crash-course/classification/roc-and-auc

[7] NIST, "Artificial Intelligence Risk Management Framework (AI RMF 1.0)", 2023. https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf

[8] Google, "Thresholds and the confusion matrix", Machine Learning Crash Course. https://developers.google.com/machine-learning/crash-course/classification/thresholding

[9] S. Kapoor, A. Narayanan, "Leakage and the Reproducibility Crisis in ML-based Science", 2022. https://arxiv.org/abs/2207.07048

Free for manufacturing and maintenance teams

Free validation plan review

Check your test set and thresholds before a model meets production.

You send

  • How images were collected, labeled, and split
  • Current thresholds and the results you have so far

You get back

  • Leakage and labeling risks we can see
  • A per-defect threshold and regression test plan

Not sure where camera inspection fits yet? A discovery call walks through your lines and assets to find the inspection and monitoring problems worth solving first. Prefer someone on site? See the free half-day on-site assessment.

Ready to put this into practice?

See how Monitory helps manufacturing teams implement these strategies.