Computer Vision Systems Case Studies
Technical5 Case StudiesUpdated Sep 2026
Computer vision work rarely fails where people expect. A model that scored well in evaluation can stall on a loading dock, a camera can drift after a maintenance visit, and a dashboard can disagree with the cameras feeding it. The five cases track teams through the gap between a working prototype and a system people rely on. Each begins with an operational problem, then separates what the team could see from what it could not, and what it chose to do first. The organizations are fictional, but the constraints and failure patterns are ones working teams will recognize.
Case Study 1
Defect detection drift after a line speed increase
Marrowgate Bottling Works, a regional beverage bottler running three filling lines with camera-based cap and fill-level inspection.
The situation
Marrowgate runs three filling lines, each with four cameras checking cap seating and fill level before cases are sealed. In the second week of the quarter, the inspection team released a retrained model that looked stronger in offline testing. Two days later, a promotional order pushed line speed up 18 percent. Frame queues on the inference servers grew, and the vision service began dropping frames under load. Operators saw cap rejects climb on line two, then line three, while the quality supervisor, Ines, found sealed cases with visible fill problems passing through. The on-call engineer had service metrics, camera logs, and an archive of rejected images, but nobody had measured the model's per-line behavior at the new speed.
The challenge
Stopping the lines would cost the plant a shipping window that its largest retail customer had already booked, and rolling back the model meant losing a fix for a scuffed-cap false reject that operators had complained about for months. Leadership wanted the speed increase kept. The team had to restore reliable detection without a full rollback, decide which line to protect first, and explain to the customer's auditors why some cases had passed inspection during the event.
Analysis
Teams grounded in Computer Vision Systems usually start by separating what the system saw from what the line experienced. Here the first useful evidence was not the aggregate reject rate, which mixed three lines running at different speeds, but the per-line confusion pattern on the archived frames. Once the team split the data by line and by frame age, the degradation looked like a timing problem rather than a model problem: frames arriving late were being scored against stale position metadata. That distinction mattered because it pointed toward a configuration fix rather than a retrain, and it made the business objective of keeping the higher speed look achievable.
The constraints were narrower than they looked. A full rollback would have restored the old behavior but also reinstated the scuffed-cap false rejects, so the team preferred a staged fix: a frame-age gate that discarded stale frames with a logged reason, plus a temporary threshold change on the two lines where rejects had spiked. Stakeholder impact was uneven. Operators needed to know which rejects to trust, the quality supervisor needed a defensible count of possibly passed defects, and the customer's auditors needed a record of which cases were affected. Each group wanted a different summary, so the team built one timeline and drew three views from it.
- 1Freeze the rolloutStop further model promotion while the team confirms which frames were affected.
- 2Split evidence by lineCompare reject and pass patterns per line, per frame age, and per speed setting.
- 3Gate stale framesDiscard frames past the age limit and log each discard with its reason code.
- 4Verify before wideningReplay archived frames at the new speed and confirm recall on known defects.
- 5Restore and monitorReturn both lines to normal thresholds and alert on frame age and reject drift.
Actions taken
- The on-call engineer, Marcus, froze model promotion and pulled per-line frame-age histograms from the shift. Line two and line three showed frames arriving well past the age limit.
- The team deployed the frame-age gate on line two first, where variance was highest. Dropped frames carried a logged reason, and operators saw a marker on any case scored without a valid frame.
- The team replayed archived frames at the higher speed and measured recall on a labeled set of known defects. Lines stayed on temporary thresholds until replay recall matched the pre-release baseline.
- The team wrote a prevention checklist requiring per-line replay tests and frame-age limits before any future speed change or model release. Alerts were added for queue depth and for reject rate drift on each line.
Results
Line two returned to its normal reject rate within the same shift, and line three followed after the replay tests passed. The team reported that the cap false rejects settled near their earlier level, while the stale-frame gate dropped a small share of frames during the peak hour. Leadership kept the speed increase for the promotional run, and the customer received a written account of the affected cases, which the auditors accepted with one follow-up sample request. The prevention checklist became a standing gate for line changes, and the next speed change went through it without a production hold.
Questions for discussion
- Which evidence would have told the team earlier that the problem was timing rather than the model, and where should that signal have been captured?
- How should the team weigh a partial fix on one line against a full rollback when the customer's commitment depends on the higher speed?
- What would a replay test need to include to be trusted as a release gate for a line running at a different speed?
Case Study 2
Conflicting pallet counts across warehouse and finance reports
Tessaly Cold Logistics, a cross-dock operator handling refrigerated freight for four regional grocers.
The situation
Tessaly operates two cross-docks where overhead cameras count pallets as trucks are unloaded and feed a nightly summary into the inventory system. In the third week of the quarter, the operations director, Lena, noticed that the warehouse dashboard showed 6 percent more inbound pallets than the finance team's receiving report for the same week. Each report was internally consistent. The camera counts came from a tracking model that assigned pallet IDs from label reads, while the finance figures came from scanner receipts that were entered when drivers signed paperwork. Two grocers had already questioned invoices, and the carrier settlement for the week was due before the discrepancy could be explained.
The challenge
The business needed a single inbound count it could defend to customers and carriers, but neither source was wrong on its own terms. Finance could not reopen the week's receipts without a formal correction process, and the vision team could not retrain the tracker without losing the pallet-level audit trail that customers had requested. Leadership wanted an answer in days, not a rebuild. The team had to find where the two numbers diverged, decide which one governed settlement, and stop the gap from recurring without freezing operations.
Analysis
Start with lineage, not with the totals. The vision count and the receipt count each passed through several transformations, and the discrepancy usually lives in one of the joins. The team traced a sample of pallets from the camera event through the tracking table, the nightly rollup, and the finance extract. The largest share of the gap came from double-counted pallets on mixed loads, where the tracker reassigned an ID after partial occlusion and the rollup did not deduplicate. Because settlement depended on receipts, the first objective was to show finance which transformed records were inflated, not to argue that the camera was right.
The constraints shaped the fix more than the diagnosis did. Finance could correct records only through a monthly adjustment, so the team proposed a reconciliation table that flagged each pallet with a match status instead of rewriting history. Customers wanted pallet-level audit trails, which ruled out simply averaging the two sources. Carriers were affected through settlement, so the team gave the carrier liaison a range with a documented basis rather than a single number that might change again. Operations staff, who had been quietly correcting counts by hand, needed to be told that the manual step was now part of the monitoring, not a substitute for it.
- Day oneOperations flags the 6 percent gap between dashboard and receiving report.
- First two daysSample pallets traced from camera event to finance extract across three stages.
- Third dayDouble-counted mixed-load pallets isolated; carrier code errors found separately.
- End of weekReconciliation table with match status published for finance and carrier liaison.
- Following monthTracker deduplication rule and daily variance alert added to the nightly job.
Actions taken
- The team froze the nightly settlement feed and published the reconciliation table with a match status for every pallet. Finance could then see which records were unverified without any retroactive edits.
- Each transformation step was given row-count and sum checks against its input, with failures written to a quarantine table instead of being silently dropped.
- The tracker's ID reassignment logic was tested on a labeled set of occluded mixed loads. The deduplication rule merged IDs only when pallet position and label reads agreed within a tolerance.
- The team added a daily variance alert between camera counts and receipts, with thresholds set by the operations lead. Manual corrections by floor staff were logged as events rather than typed into totals.
Results
Finance reconciled the affected week with flagged records rather than edits, and the carrier settlement went out on time using the documented range. The double-counting rule removed most of the mixed-load gap, leaving a residual difference that the team traced to a single carrier code and corrected at source. Over the following month, the daily variance alert fired four times, and each alert led to a fix in the feed rather than a manual override. Customers received the pallet-level trail they had asked for, and the operations director stopped receiving separate versions of the same weekly figure.
Questions for discussion
- Which lineage checkpoints should exist before a camera-derived count is allowed to feed a financial report, and who should own each one?
- When two internally consistent sources disagree, what rule should decide which one governs a settlement, and where should that rule be written down?
- How would the team know that a variance alert threshold is too loose or too tight after the first month of operation?
Case Study 3
Possible compromise of in-store camera analytics gateways
Lumen Harbor Grocers, a regional grocery chain with 34 stores running edge gateways for shelf and queue analytics.
The situation
Lumen Harbor runs edge gateways in each of its 34 stores, and each gateway runs a vision model that counts shoppers, measures queue length, and flags empty shelf facings. In the second week of the quarter, the security analyst, Tomas, noticed that one gateway had opened an encrypted connection to an address outside the vendor's published range, at an hour when the store was closed. The vendor's remote maintenance log showed a support login from an unfamiliar network two days earlier. The gateway also held several hours of retained frames, which it normally clears within minutes. The cause was unknown: vendor error, stolen credential, or altered model package.
The challenge
Stores are open to customers, so the team could not simply pull power from 34 gateways without losing queue data and shelf alerts the stores depended on. Evidence was also time-sensitive: retained frames contain shoppers, and privacy counsel needed to know who had access before anyone copied them. The vendor wanted to push an update immediately, which would overwrite the state that investigators needed. The team had to scope which gateways were affected, preserve evidence without over-collecting personal images, contain the connection, and record each step as it happened.
Analysis
Security response in a camera system starts with a question about what could have left the device, and the answer depends on what the device kept. The gateway's retained frames were the most important evidence and also the most sensitive, so the team treated them as a restricted artifact: hashed on the device, copied once to a sealed store, and logged with each access. Network flow records from the store router and the vendor tunnel gave a timeline of outbound sessions. Model package checksums showed whether the inference binary matched the vendor's release. Together these sources let the team separate the three explanations without touching live store operations.
The containment choice was a trade between two kinds of harm. Blocking the outbound address on all gateways would have been fast, but a store-wide change could also cut off queue alerts that staff used during peak hours, and it would warn a possible attacker. The team instead isolated the one suspect gateway onto a restricted network segment, revoked the vendor's remote maintenance credential, and held a rollback to the last verified model package. Stakeholder impact fell unevenly: store managers needed to know their shelf data might be delayed, and the vendor had to accept that its update would wait until the evidence was preserved.
Actions taken
- The analyst, Tomas, opened an incident record and set a legal hold on the gateway's frame store, noting who had access and when. Only hashed copies left the device, and the originals stayed sealed.
- The suspect gateway was moved to a restricted segment with outbound traffic limited to the security team's collection host. Store operations continued on a local fallback that kept shelf alerts running.
- The vendor's remote maintenance credential was revoked, and the vendor supplied session records for the suspicious login. Model checksums on every gateway were compared against the release manifest.
- Findings were written as a sequence of steps with the authority for each, including who approved isolation and who approved the held update.
Results
The investigation found that the vendor's support credential had been reused on a personal device, and that the gateway's outbound session carried a diagnostic bundle rather than shopper frames. Checksums matched the release manifest on 33 of 34 gateways, and the one mismatch was the isolated device, which was rebuilt from the verified package. The retained frames were reviewed under counsel's direction and deleted after the hold ended. Store managers reported delayed shelf data for less than a day, and the vendor issued new credentials under a tighter access policy.
Questions for discussion
- What would have changed in the response if the gateway had retained no frames at all, and how should retention limits be set before an incident?
- Where is the line between isolating a device and warning a possible attacker, and who should make that call at two in the morning?
- How should a vendor's remote access be authorized, logged, and reviewed so that its records can be trusted during an investigation?
Case Study 4
Choosing where grading models run across packing houses
Pardee Orchard Imaging, a company that grades fruit with line cameras for 58 packing houses across four regions.
The situation
Pardee grades fruit with cameras mounted over packing-line conveyors. Each line sends frames to a central inference service, which returns a grade within a fixed window before the fruit reaches a diverter. The service began as a single regional deployment and now covers 58 packing houses. During the peak season, a new customer added two lines per site and roughly doubled frame volume in a month. Median response time stayed acceptable, but the tail latency at the 99th percentile crossed the diverter timeout on several lines, and the monthly hosting bill rose faster than contracted volume. Packing managers wanted a decision before the next harvest, and the engineering team had three options on the table.
The challenge
Every option moved a constraint rather than removing one. Cloud-only inference kept model updates simple and security review in one place, but it depended on the packing-house network, which was unreliable in two rural sites. Edge inference on line PCs cut latency but meant maintaining model versions on hardware that the company did not own and that customers could reboot at will. The business objective was a stable grade within the timeout at a cost per frame the contract could carry, and the team needed measurements from real sites.
Analysis
The strongest evidence came from the packing houses themselves, not from the benchmark. The team pulled three weeks of timing logs from fourteen sites and separated network time, queue time, and model time for each frame. Model time was stable across sites, so the tail problem sat mostly in queueing and in retries after dropped connections. That finding changed the objective: the question was no longer which location is fastest, but how much of the timeout budget each option could protect under a bad network day. Cost per frame was measured separately, since the hosting bill was driven by peak concurrency rather than average volume.
Security and operations pushed in different directions. Keeping frames on site reduced data movement and simplified a customer's data residency requirement, but it meant every line PC needed signed model packages, a rollback path, and remote health checks that the operations group had never run at this scale. A hybrid design kept the fast first grade on the line and sent only ambiguous frames to a central service for rescoring, which lowered cloud calls by a share the team could bound from the logs. Packing managers cared most about the diverter timeout and about not losing a shift to an update.
- Measure site timing Split each frame's time into network, queue, and model components per site.
- Model options Price cloud, edge, and hybrid against the timeout budget and peak concurrency.
- Pilot two sites Run the chosen placement on sites with the worst network and the highest volume.
- Review drift Compare grades and timeouts after each season, then restart the measurement.
Actions taken
- The team pulled timing logs from fourteen sites and built a per-frame breakdown of network, queue, and model time. Results were reviewed with two packing managers before any design work began.
- A hybrid prototype ran on one site's line PC: the edge model graded every frame, and frames near the grade boundary were sent to the central service for a second opinion.
- Model packages were signed, versioned per line, and rolled out in waves of two sites, with automatic rollback when the health check failed twice in a row.
- Cost was tracked per thousand frames and per concurrent connection, and a budget alarm was set on the hybrid path. Edge hardware was limited to models that could roll back within one shift.
Results
The hybrid design brought the 99th percentile of grade time under the diverter timeout on all pilot lines, including the two rural sites where the cloud-only option had failed. Cloud calls fell by roughly four in ten frames, which lowered the hosting bill below the contracted volume trend. The team did not claim the design was cheaper in every case, since the edge hardware had a per-line cost the packing houses had to accept. The first rollout wave completed before harvest, and one rollback in the second wave took under ten minutes.
Questions for discussion
- Which measurements would have justified moving to the hybrid design earlier, and which ones would have been misleading if taken from a benchmark?
- How should a team decide how much edge hardware it can responsibly support when the hardware belongs to the customer?
- What rollback time should be treated as the acceptable limit for a model change on a live production line, and who should agree to it?
Case Study 5
Automated damage triage before adjuster review
Calder Ridge Mutual, a regional auto insurer that routes vehicle claims through photo review and adjuster teams.
The situation
Calder Ridge Mutual handles vehicle claims for several regions, and each claim begins with photos sent by the policyholder or repair shop. Adjusters spend the first part of each file sorting photos by damage type. A vendor proposed a vision model that would classify damage severity from photos and route claims to fast-track approval, claiming a large cut in handling time. The internal pilot covered about 9,000 claims over a quarter, but most of them came from two urban regions where photo quality and vehicle mix were similar to the training data. Rural claims, older vehicles, and shops that used phone photos in poor light were thinly represented, and the vendor report had no breakdown by any of them.
The challenge
The efficiency case was real on paper, and leadership wanted the handling savings in the next budget cycle. The risk was that a model trained on the wrong mix of photos would quietly under-estimate damage on the claims least likely to be checked, and those claims often belonged to policyholders in rural areas or older vehicles. The team needed metrics that reflected claim outcomes rather than classification accuracy alone, human oversight that did not simply rubber-stamp the model, and a rollout that could be stopped within a day.
Analysis
Start by deciding what the pilot can and cannot show. Overall accuracy on the pilot looked strong, but the pilot's sample was concentrated where the model was most likely to be right. The evaluation the team designed reported error by segment (region, vehicle age, photo source, and repair shop) and tracked two outcome metrics: the rate at which fast-tracked claims later needed reopening, and the rate at which the model's severity estimate differed from the adjuster's final estimate by more than a set margin. Those measures matter because they connect to money and to fairness, while classification accuracy only describes the model.
Human oversight was the constraint that decided the design. Adjusters were already stretched, and a review step that asked them to re-check every fast-tracked claim would erase the savings. The team instead set a sample rate by segment, so that thinly represented groups were reviewed more often than urban claims with a long history of good photos. Policyholders were affected by any error in either direction, and repair shops had a stake in how quickly they were paid. The vendor, meanwhile, had a contract tied to handling-time savings, which gave it little reason to highlight segment gaps unless the insurer asked.
Actions taken
- The team defined the evaluation before the next pilot: error by segment, reopen rate, and severity gap thresholds were written into the test plan with agreed limits.
- A stratified sample of past claims was relabeled by senior adjusters, with rural and older-vehicle files deliberately oversampled to test the model where it was weakest.
- The model was deployed to one region in shadow mode first, where its severity estimates were logged beside adjuster decisions without changing any claim outcome.
- Fast-track authority was capped by claim value and segment, with an automatic pause if reopen rates or severity gaps crossed the agreed limits for two consecutive weeks.
Results
The shadow phase showed that the model's severity estimates were close to adjusters on urban claims but drifted on older vehicles, where the gap crossed the agreed limit in the third week. The fast-track authority was limited to small urban claims in one region, and the other segments stayed under review. Handling time fell for the fast-tracked group by a margin the finance team accepted, while the rural queue did not change. The vendor was asked to retrain with the oversampled files, and the insurer kept the automatic pause in its runbook.
Questions for discussion
- Which segments should be oversampled in a pilot, and how would the team know its sample design had hidden a weakness it was not looking for?
- How much review should a human keep once a model has performed well for a full season, and what evidence would justify reducing it?
- Who should hold the authority to pause a fast-track rule, and what should happen to claims already approved during the window before the pause?
Learning Resources
Technical CASE STUDIES
China, India, United States, Indonesia, Pakistan, Nigeria, Brazil, Bangladesh, Russia, Japan, Mexico, Philippines, Vietnam, Egypt, Turkey, Germany, United Kingdom, France, Italy, Canada, South Korea, Australia, Spain.