I entered this project with limited guidance and quickly learned that training a computer-vision model was only part of the problem. The harder problem was determining whether the data and evaluation process actually represented production performance.
One of the first concepts I learned was mean average precision. mAP50 evaluates detections using an intersection-over-union requirement of 0.50. mAP50-95 averages performance across increasingly strict overlap thresholds between 0.50 and 0.95. I initially treated improvements in those metrics as direct proof that the model itself was improving.
That assumption was incomplete. A metric can only describe the validation data provided to it. If the annotations are inconsistent, the class distribution is unbalanced, or the validation examples are extremely similar to the training examples, the resulting score can look excellent without representing real-world generalization.
I manually worked through approximately 840 videos and created more than 4,000 surface annotations. At the time, I believed every visually clear frame was valuable. If a video contained 30 clean frames of the same surface, I might annotate all 30.
Those adjacent frames were often nearly identical. They showed the same physical object, angle, lighting, background, and capture session only milliseconds apart. The dataset contained many annotations, but it did not contain the amount of independent visual diversity that the raw count suggested.
Aggressive augmentation helped compensate for some of those weaknesses. Through a combination of volume, augmentation, and a considerable amount of luck, I created an amalgamation of a model that appeared to work roughly 95 percent of the time during the scenarios I was testing.
The model worked well enough to be useful, but the foundation was difficult to understand. When a change improved the metrics, I could not always determine whether the model had genuinely learned something new or had simply become better at recognizing variations of examples it had effectively already seen.
That was where I learned the importance of data engineering and data sanitization. In a production environment, completely discarding a working system is not always an immediate option, so I began developing two versions in parallel: a production hotfix and a complete long-term solution.
The hotfix removed the weakest data, corrected poor annotations, and retained roughly 2,000 of the most useful existing frames. I then added approximately 100 deliberate examples for each surface. This improved representation across different object models while preserving enough high-quality segmentation examples to maintain mask precision.
That version stabilized production and created enough time to rebuild the dataset correctly.
After explaining the importance of a clean dataset to leadership, I was given control over the annotation workflow. I drafted an approximately 25-page work-instruction document defining class boundaries, surface rules, overlap handling, rejection criteria, edge cases, and examples of correct and incorrect annotations.
I trained the annotation team and established a feedback loop for ambiguous examples. When a new edge case appeared, it was reviewed, resolved, documented, and incorporated into the instructions so the same uncertainty would not continue spreading through the dataset.
Even with written instructions and a trained team, I did not assume the finished labels were correct. I created a lightweight review script that displayed annotated frames rapidly and allowed each example to be classified as acceptable or requiring additional review.
I manually reviewed approximately 3,700 annotations. Around 340 were identified for review, and I returned to those examples and corrected each one myself. At that point, I had a dataset whose annotation quality I could actually trust.
Once the dataset and validation split were clean, accuracy, precision, recall, F1 score, and mean average precision became much more meaningful. The confusion matrix also became easier to interpret. Correct predictions should concentrate along the diagonal, while remaining off-diagonal values expose specific classes the model is confusing.
The cleaner organization created a reliable development loop. I could add a controlled set of examples, retrain, inspect the metrics, review the failures, and understand why the result changed instead of guessing which part of the dataset had influenced it.
The next stage focused on diversity rather than raw volume. I began capturing roughly 450 targeted photos for each surface across different lighting conditions, object colors, backgrounds, viewing angles, and capture distances. I also investigated carefully controlled synthetic augmentation against realistic backgrounds.
The goal was no longer to make the dataset larger as quickly as possible. The goal was to make each new image contribute information the model had not already seen.
The biggest lesson was that computer-vision performance is inseparable from data engineering. A complex model trained on poorly organized data can produce impressive-looking metrics while remaining unpredictable. A clean, diverse, manually reviewed dataset makes both the model and its evaluation understandable.