Computer Vision Systems Best Practices

01Purpose and scope

This manual supports the course Computer Vision Systems. It is written as a working reference for engineers, technical leads, and analysts who are responsible for getting a system that reads images or video into production and keeping it useful after launch. It covers the path from defining the visual task through data collection, model selection, integration with cameras and downstream software, and ongoing monitoring.
The scope is deliberately practical. It does not teach the mathematics of convolutional networks or transformer architectures in depth, since those topics are covered in standard texts and courses. Instead, it describes how the work is actually organized: who decides what the system should see, how images are gathered and labeled, how results are checked against the real operating conditions, and what happens when lighting, camera position, or product appearance changes.
Computer vision systems appear in many settings. A quality inspection line may flag scratched parts, a retail store may count shoppers at a door, a warehouse may read package labels, and a clinic may pre-screen images for a radiologist to review. The engineering concerns overlap across these cases, but the acceptable error rate, the cost of a missed detection, and the legal constraints on collecting imagery differ a great deal. This manual points out those differences where they change the practice.

02Core principles

Start from the decision the system supports, not from the model. A vision system is useful only when its output changes an action: a part is rejected, a door alarm is raised, a human reviewer is notified. Write down the action, the person or process that takes it, and the cost of each type of error before choosing a model or camera.
Treat the data as the specification. In most projects, the images and labels define what the system will actually learn to do. If the training set contains only daytime images from one camera, the system's behavior at night or from another angle is unknown no matter how sophisticated the network. Record how each image was captured, under what conditions, and by whom.
Measure on data that resembles production. Accuracy on a randomly held-out split from the same batch of images often overstates real performance. Keep a test set collected from later dates, different shifts, or different sites when possible, and report results separately for each condition that matters.
Design for failure. Cameras get dirty, mounts drift, firmware updates change compression, and products get new packaging. Plan for the system to report low confidence or to pass uncertain cases to a person rather than silently guessing.

03Planning and preparation

Begin with a short task definition. Name the objects or events to detect, the output format (class labels, bounding boxes, segmentation masks, counts, or text), the required frame rate or latency, and the place where results will go. A clear one-page definition prevents the common situation where the vision team and the operations team each assume a different meaning for the word defect.
Inventory the imaging setup before collecting data. Record camera model, lens, resolution, shutter type, frame rate, field of view, mounting height, lighting type, and expected ambient conditions. Note any constraints from the plant floor or building, such as network access, power at the mounting point, or restrictions on where cameras may be placed. For industrial robot cells, safety requirements in ISO 10218 and the local machine safety regulations apply, and the integration team should confirm them with the responsible safety engineer.
Plan the data budget. Estimate how many images of each class you need, how rare the critical events are, and how long it will take to capture enough examples of them. Rare failures are often the hardest part of the project. If a defect appears once in several thousand parts, you may need a deliberate collection plan, controlled defect samples, or synthetic augmentation, and each of those choices has limits that should be stated in the plan.
Confirm privacy and data handling before any recording starts. Footage that shows people, license plates, or identifiable spaces may fall under privacy laws such as the General Data Protection Regulation in Europe or state-level rules in the United States. The rules depend on jurisdiction and change over time, so check the current requirements with your legal or privacy office and document the decision.

04Recommended workflow

1
: Define the labeling guide. Write a document with examples of each class, borderline cases, and explicit rules for occlusion, partial objects, and ambiguous lighting. Have two labelers annotate the same sample set and compare results. Disagreements usually reveal gaps in the guide, not careless labelers.
2
: Capture and organize the data. Record images with metadata for camera ID, timestamp, location, operator, and lighting setting. Store raw files unmodified. Split the data into training, validation, and test sets in a way that keeps near-duplicate frames from the same video segment in one split only, otherwise the test score will be inflated.
3
: Establish a baseline. Before training a custom model, try a pretrained backbone with a simple head or a classical method such as thresholding and blob analysis if the task is constrained. A baseline shows what is achievable with little effort and gives the project a reference point for cost and complexity.
4
: Train and tune. Adjust the model architecture, input resolution, augmentation, and learning rate against the validation set. Augmentations should reflect real variation, such as brightness shifts or small rotations, and should not create conditions that never occur on the line. Log every run with its data version, configuration, and random seed.
5
: Evaluate on held-out conditions. Score the model on the test set and on any separate site or time-based set. Compute precision and recall for each class, not only overall accuracy. For object detection, report mean average precision at the intersection-over-union threshold agreed in the task definition. Inspect the worst false positives and false negatives by hand.
6
: Integrate and test in the target environment. Run the model on the actual camera stream, on the actual hardware, with the actual network path. Measure end-to-end latency, including capture, preprocessing, inference, and the message sent to the programmable logic controller or database. A model that is fast on a workstation may miss its deadline on an embedded device.
7
: Deploy in shadow mode first. Let the system produce predictions while humans continue to make decisions. Compare the two over several weeks of operation, or over the period set in the plan, and review disagreements. Only move to automated action when the agreed thresholds are met and the operations owner signs off.
8
: Set up monitoring and a retraining trigger. Define the signals that show drift, such as a shift in average confidence, a change in class distribution, or rising reviewer overrides. Decide in advance what level of change triggers an investigation and what level triggers retraining.

05Quality checks, risk controls and compliance

Quality checks should cover the data, the model, and the deployed system. For data, check label consistency, class balance, image quality (blur, exposure, dead pixels), and whether the test set has been contaminated by training frames. For the model, check calibration of confidence scores, performance by subgroup or condition, and robustness to small perturbations such as added noise or slight camera shift. For the deployed system, check timing, memory use, dropped frames, and the behavior after a camera restart.
Risk controls depend on what happens when the system is wrong. A false alarm on a conveyor may cost a few minutes of line time. A missed defect in a pharmaceutical blister pack may reach a patient. Classify each use by consequence, then set required controls: human review for high-consequence decisions, dual sensors for critical stops, or a fail-safe default when the camera feed is lost. Controls that depend on a single unverified model output are rarely adequate for high-risk work.
Regulated and high-risk fields need extra care. In medical imaging, a vision system may fall under medical device regulation and require authorization before clinical use. In transportation, safety functions may be subject to specific standards and certification. In finance, identity verification with face images may trigger anti-fraud and consumer protection rules. This guidance does not replace qualified professional judgment, approved procedures, or required authorization from the responsible regulator or internal safety authority.
Bias and fairness deserve explicit checks where people are detected or identified. Measure performance across skin tones, clothing, age ranges, or other attributes relevant to the setting, and document the results. Where the law restricts biometric processing, follow the rules for your jurisdiction and get legal advice before deployment.

06Collaboration and documentation

Assign roles early. A typical team includes a product or operations owner who defines the decision, a vision engineer who builds and tunes the model, a data lead who manages labeling and storage, an integration engineer who connects the system to cameras and controls, and a quality or safety representative who approves the risk controls. In small teams one person may hold several roles, but the sign-off responsibilities should still be written down.
Keep a model card or system record for each release. It should state the intended use, training data sources and dates of capture described in general terms, known failure conditions, evaluation results by condition, thresholds, hardware and software versions, and the owner who approved deployment. Future engineers need this record to understand why the system behaves as it does.
Use version control for code, labeling guides, configuration files, and dataset manifests. Large image datasets are often stored in object storage with a manifest file that lists checksums and metadata. Tie every deployed model to the exact dataset and configuration that produced it so that a regression can be traced.
Hold regular review meetings with operators. The people who watch the line or the screen often notice problems first, such as a new reflection on a conveyor or a camera that has rotated after maintenance. Give them a simple way to flag cases and to see what happened to their flags.

07Measures and continuous improvement

Choose a small set of indicators that connect to the decision. Common ones include precision and recall for critical classes, false alarm rate per shift, end-to-end latency at the ninety-fifth percentile, percentage of frames processed without drops, share of cases escalated to human review, and reviewer override rate. Avoid a single headline accuracy number, since it can hide a serious failure in a rare but costly class.
Review the indicators on a fixed schedule and after any change to the camera, lighting, product, or software. Compare current values with the shadow-mode baseline and with the agreed thresholds. When performance drops, look first at the input data, because most field failures come from changed conditions rather than from the model code.
Improve through a loop of collection, labeling, and retraining. Capture images of the failures, add them to the labeled set with the same labeling guide, retrain, and evaluate on the same held-out conditions plus the new ones. Keep the old test sets fixed so that improvements are real and not a result of changing the yardstick.
Know the limits. Some problems cannot be solved with more data from the same camera, such as a view that is occluded by machinery or a lighting setup that washes out the target. In those cases the right action is a hardware change, a new sensor, or a narrower scope for the system.

08Common mistakes and how to prevent them

Training on data that is too clean. Images gathered during a calm week in good lighting rarely reflect the full range of operation. Prevent this by sampling across shifts, weather, seasons, and maintenance periods, and by including the messy cases on purpose.
Leaking test data into training. Frames from the same video, or the same part photographed several times, can appear in both sets. Prevent this by splitting by episode, session, or production lot rather than by individual image.
Ignoring the label definition. When labelers disagree about borderline cases, the model learns noise. Prevent this with a written guide, periodic calibration sessions, and an agreement measure reported for each batch.
Trusting confidence scores without calibration. A score of 0.9 may not mean the object is present nine times out of ten. Prevent this by plotting reliability curves and choosing thresholds from the validation data rather than from a default value.
Skipping the field test. A model that works on a workstation with saved images may fail on a live stream because of compression, buffering, or timing. Prevent this with end-to-end tests on the target hardware before any decision depends on the output.
Forgetting maintenance. Cameras drift, lenses collect dust, and lighting bulbs dim. Prevent this with scheduled checks, reference targets in the scene, and alerts on image statistics such as average brightness or sharpness.

09Practical checklist

Write a one-page task definition that names the decision, the output, the latency limit, and the owner who acts on results.
Document camera, lens, lighting, mounting, and environmental conditions before collecting data.
Confirm privacy, safety, and data retention requirements with the responsible office for the jurisdiction.
Create a labeling guide with borderline examples and run an inter-labeler agreement check.
Split data by session, lot, or site so near-duplicate frames never cross between training and test sets.
Build a simple baseline and report it alongside the custom model.
Evaluate precision and recall per class and on each operating condition that matters.
Test end-to-end latency and frame handling on the actual deployment hardware.
Run the system in shadow mode and review disagreements with the operators before automating any action.
Keep a model record that lists versions, data lineage, thresholds, known limits, and approvals.
Define drift signals and a retraining trigger, and review them on a fixed schedule.
Confirm fail-safe behavior for camera loss, network failure, and low-confidence output.

★Learning sources

Micro Degree: Computer Vision Systems
Micro Degree: Data Analytics Applications
Micro Degree: Azure Cloud Fundamentals
Micro Degree: Google Cloud Platform Basics

?Frequently asked questions

How much labeled data do I need before starting?

The amount depends on the number of classes, the visual variation in the setting, and how rare the critical events are. A useful first step is to collect a small pilot set, label it with the guide, and measure how performance changes as you add more examples. Plan for more data than you expect, and budget time for capturing the rare cases.

Should I train my own model or use a pretrained one?

For most industrial and commercial tasks, starting from a pretrained backbone and fine-tuning it on your own images is faster and needs less labeled data than training from scratch. A simple classical baseline is still worth trying first when the scene is tightly controlled. The choice should be justified by the measured results on your test conditions, not by preference.

How do I know when the system is ready to make automated decisions?

Run it in shadow mode long enough to observe the common conditions and several of the rare ones, then compare its output with human decisions. Automated action should begin only after the agreed precision, recall, and latency thresholds are met and the operations owner and safety or quality representative approve the change. In regulated or safety-critical settings, the required approvals and any certification steps come from the responsible authority, and this guidance does not replace them.