Machine Learning and Artificial Intelligence Engineering Case Studies
Technical5 Case StudiesUpdated Sep 2026
These five case studies follow machine learning and AI engineering work from the moment a model begins making decisions for customers, staff, or machines. Each comes from a fictional organization and is told from the seat of the engineer who had to act. The situations cover a serving incident, a reporting conflict, a suspected compromise, a scaling decision, and a screening tool evaluation. Track four threads through all of them: design constraints, operating data, quality tolerances, and maintenance records. Every one shows what the team knew, what it assumed, and what it changed.
Case Study 1
Release traffic spike and a stalled ranking service
Tarrow Market, a regional online grocery marketplace with an in-house product ranking service
The situation
Tarrow Market released a new ranking model to its product listing pages on a Thursday afternoon, two days before a weekend promotion it had advertised for several weeks. The model had passed offline evaluation and a canary at five percent of traffic. Within the first hour of the promotion, the ranking service's p95 latency climbed from about 140 milliseconds to over 900, and roughly one in twelve listing requests returned the fallback order, which was sorted by price and no longer personalized. Request volume peaked near 4,800 per minute, nearly three times the canary level. Search-to-cart conversion fell sharply that evening, and the on-call rotation was paged for the third time that week.
The challenge
The release was not the only change in that window. A feature store backfill had been scheduled for the same evening, and nobody had linked it to the release calendar. The fallback path reads from a cache sized for traffic from before the last catalog expansion, so every miss went straight to the feature store. The on-call engineer, Priyanka, had three options: roll back the model, scale the feature store, or turn off personalization entirely. Each carried a different cost to revenue, and each risked hiding the real fault.
Analysis
Teams grounded in Machine Learning and Artificial Intelligence Engineering tend to read design constraints before they read dashboards. Here the constraints existed but were scattered: the latency budget for listing pages was 250 milliseconds, the fallback path had never been load tested beyond canary volume, and the feature store's connection pool limit was known only to the data platform group. The operating data became clearer once it was split by path. Cache hit rate, feature store queue depth, and fallback rate moved together, while the model's inference time barely changed. That pattern pointed at the data path, not the model.
Quality tolerances were the other half of the picture. The team had agreed that fallback responses could be served for no more than two percent of listing requests, and that p95 latency above 400 milliseconds for longer than ten minutes would trigger a rollback review. The runbook still referenced the old cache cluster. Maintenance records settled the question of cause. The backfill job had been added to a shared schedule eleven weeks earlier with the note 'run nightly', and the cache sizing ticket had been closed without a load test. Reading those records first saved the team from a rollback that would have changed nothing.
- Detect the drift Latency and fallback rate were split by data path, not model path.
- Diagnose by path Cache hits, queue depth, and fallback rate were compared before any rollback decision.
- Restore capacity first Cache resized and connection pool raised within forty minutes.
- Prevent with checklists Batch schedules, cache capacity, and traffic forecasts reviewed before every model release.
Actions taken
- Priyanka expanded the cache to the size the model team had originally specified and raised the feature store connection pool limit, which cleared most fallback traffic within twenty minutes.
- Kept the model serving, since its inference time was never the problem, and froze the nightly backfill until its load estimate had been checked against promotion traffic.
- Added a reconciliation step to the release checklist that compares planned batch jobs, cache capacity, and expected traffic before any model change ships.
- Rewrote the runbook thresholds with their owners' names attached, linked each alert to a dashboard panel, and ran a load test at twice the forecast peak before the next promotion.
Results
Service recovered within about forty minutes of the page. Fallback rate fell below one percent by the end of the first evening, and p95 latency returned to the mid-150 millisecond range after the pool was resized. The model was left in place and later showed a modest lift in add-to-cart rate across two weeks. The promotion finished with conversion about nine percent below the prior weekend, a loss the finance group estimated at 38,000 in gross margin. The backfill was rescheduled behind a capacity gate, and the release checklist has caught two similar conflicts since.
Questions for discussion
- Which constraint in the team's documentation would you have checked first, and what evidence would have pushed you to check it sooner?
- How would you decide between a rollback and a capacity change when the model itself looks healthy?
- What belongs in a release checklist for a model that depends on batch jobs and caches owned by other teams?
Case Study 2
Conflicting load reports traced to snapshots and gap filling
Ferrow Grid Services, a regional electricity distributor whose forecasting group supplies demand models to operations and finance.
The situation
Ferrow Grid Services produces a weekly demand forecast for each of its substations, and the operations and finance groups both use it. During a cold spell, the two reports disagreed by about 11 percent for the northern service area. Operations had planned crew coverage around the higher number, while finance had booked a revenue estimate around the lower one. The forecasting team, which has nine people and one data engineer shared with billing, was asked to explain the gap before the next planning meeting. The disagreement was not in the model. Both reports consumed the same forecast table, but they applied different filters and different joins before the figures were reached.
The challenge
The team had a lineage diagram that stopped at the forecast table. Nobody had documented the transformations between the table and the two reports, and two analysts had each written their own join. Meter readings from three substations arrived late or with gaps, and the ingestion job filled those gaps with zeros in some paths and with the previous day in others. The question was not which report was right. It was which rules each report was actually using, and whether either matched what planning believed.
Analysis
The design constraints here were simple to state and hard to honor. The forecast had to be reproducible for any past week, which meant every transformation needed a version and an input snapshot. Operating data showed that the two reports had been generated from different snapshots of the meter table, one taken before the late readings arrived and one after. Comparing row counts per substation per hour exposed the mismatch quickly. The northern area had 312 hourly rows missing from one snapshot, and the gap was not random. It lined up with the three substations whose telemetry had the longest reporting delays.
Quality tolerances had never been written down for this table. The planning group could live with a forecast that was off by up to 3 percent in a single area during a cold spell, but a gap wider than that needed a flag and a human check. Maintenance records explained why the tolerance was undefined. The gap-filling rule was changed by a developer whose change lived only in a commit message. The team now records each tolerance as a test with an owner, a threshold, and a link to the report it protects, so that the next disagreement starts from a rule rather than a memory.
Actions taken
- Pinned each report to a named snapshot of the meter table and stored the snapshot identifier in the report footer, so either figure could be regenerated exactly.
- Replaced the two hand-written joins with a single shared transformation module, reviewed line by line by both analysts and tested against 40 known weeks.
- Set gap handling to flag missing hours rather than filling them, and routed any substation with more than six hours of delay to a manual check before publication.
- Added row count and value range checks per substation per hour, with alerts sent to the forecasting channel and to the owner of each upstream meter feed.
Results
The two reports were reconciled within the week, and the gap closed to under one percent once both used the same snapshot and gap rule. Planning kept its crew coverage plan, and finance restated its revenue estimate with a footnote naming the snapshot. Six weeks later, a sensor outage at one substation triggered the new manual check, which caught a stale feed before it reached a published forecast. Monitoring now produces a weekly list of tolerance breaches, and the most recent run listed four entries, all resolved before planning met. The monthly reconciliation now starts from a shared baseline.
Questions for discussion
- Which of the two reports would your team have trusted first, and what in the lineage would have changed that instinct?
- How should a tolerance for a forecast be set when planning and finance accept different levels of error?
- What should happen to a gap-filling rule when the person who wrote it is no longer on the team?
Case Study 3
Suspicious model registry activity and a possible credential leak
Corliss Claims Services, a claims unit inside a regional insurer that runs automated triage models for property losses.
The situation
During an overnight window, the security team at Corliss Claims Services noticed a service account pulling model artifacts from the registry at a rate the account had never shown. Fourteen artifacts were downloaded in under six minutes, from an address with no prior registry history. The account belonged to a scheduled evaluation job, and its token had been committed to a configuration file in a repository that more than 200 contractors could read. The triage model scores property claims, and some of its training samples contain policyholder addresses and loss narratives. The on-call analyst, Teodor, had to decide what to contain before anyone knew whether the token was actually misused.
The challenge
Preserving evidence and containing risk pulled in opposite directions. Revoking the token would stop further downloads but would also break the evaluation job, which feeds a retraining pipeline due in the next two days. Rotating the registry key would be cleaner but would touch services owned by three other teams. Logs were retained for a limited window, and the registry's audit export ran only once per day. Every hour spent deciding was an hour in which the artifacts could leave the company.
Analysis
Design constraints shaped the response more than the threat did. The registry had been built for speed of promotion between environments, so artifacts were readable by any service account with a registry role, and there was no per-artifact access log. The operating data came from two places: the registry's request log, which recorded account and source address, and the job scheduler, which showed that the evaluation job had not run at the time of the downloads. That mismatch was the strongest early signal. A scheduled job that is not running cannot be the caller, so either the credential was in use elsewhere or the scheduler's record was incomplete.
Tolerances in security work are lopsided. A false alarm costs an afternoon of analyst time, while a missed exfiltration may carry notification obligations and regulatory exposure. The team had a written tolerance for rotation outages, two minutes of downtime per service, but nothing for evaluation jobs. Maintenance records showed why. The token had been created eighteen months earlier with a comment reading 'temporary, remove after pilot', and the pilot's ticket had been closed without removing the credential. Reviewing that history turned an open question about intent into a clear finding about a missed cleanup step.
Actions taken
- Revoked the token and issued a replacement scoped to the evaluation job alone, after the team had exported the registry logs and snapshotted the affected artifacts for the record.
- Froze the registry's read access for non-essential service accounts and placed the fourteen downloaded artifacts under a hold, with hashes recorded in the incident ticket.
- Asked the evaluation owner to confirm, in writing, every run from the past month, and checked each run's inputs against the scheduler and the source address list.
- Moved the token into a managed secrets store, scrubbed the repository history, and added a pre-commit scan that blocks credential patterns before they reach the shared remote.
Results
Containment was in place before the morning review, and the incident was closed as a credential exposure with no confirmed exfiltration after the evaluation owner's log review and the registry export matched the scheduler record. The source address turned out to belong to a contractor's build agent that had been reused from an older project, which explained the downloads and why no alert had fired before. Two evaluation runs had to be repeated because their artifacts were under hold. The authorized response steps were written into the security runbook, and registry roles now get a quarterly review.
Questions for discussion
- At what point in this response would you have accepted a larger operational cost in exchange for cleaner evidence?
- Which of the tolerances described here should belong to the model team, and which should belong to security?
- How would you tell an expected job from an impersonator when the scheduler's own record is incomplete?
Case Study 4
Choosing a regional inference design for a translation product
Quarry Bay Translations, a document translation product sold to legal and medical firms with growing data residency requirements.
The situation
Quarry Bay Translations runs a translation service whose inference fleet sits in one cloud region. Enterprise customers in three additional jurisdictions have begun requiring that source documents and model outputs stay within their own borders, and two of them have made it a contract condition. Average request size has grown to roughly 6,500 tokens, up from about 2,100 when the current architecture was designed. Short sentences remain fast, but long legal documents now queue behind each other during business hours, and p99 has crossed the ten-second ceiling written into the enterprise service agreement. Monthly inference spend has risen by more than a third.
The challenge
The team has three realistic options and a short window to choose. A second region with full replicas would satisfy residency but double the fixed cost and add a synchronization problem for model versions. Per-customer deployments would isolate data cleanly but would multiply the number of model instances the operations group must patch. Routing long documents to a larger GPU tier would fix latency but not residency. Any choice has to hold up against the security review that the two largest customers have scheduled.
Analysis
Start with the constraints that cannot move. Residency is a legal boundary, so any design that lets source text leave a jurisdiction is excluded. The latency ceiling is contractual, which means p99 matters more than the median. Operating data shows where the time goes: on long documents, more than half of the wall clock is spent waiting in the queue rather than generating tokens, because batches are filled by arrival order and a single 40-page filing can hold up a dozen shorter jobs. A larger GPU tier attacks generation time, which is a minority of the delay, while batching by length attacks the queue directly.
Tolerances matter here too. The translation quality gate samples 2 percent of outputs for review by bilingual reviewers, and any drop in the sampled score below the agreed threshold blocks a model version from promotion. A second region would need its own sampling pipeline. Maintenance records argue against the per-customer option. The last three model upgrades each took an average of nine working days to patch across the existing fleet, and two of those upgrades left a stale tokenizer on one instance for a week. Running more instances means more of these records, not fewer, and the tolerance gate covers only what it can sample.
- 1Jurisdiction checkEach request is tagged with the customer's residency rule before routing.
- 2Length bucketingLong documents get their own queue, so shorter jobs are not held.
- 3Regional inferenceSource text and generated output stay inside the jurisdiction's cluster throughout.
- 4Sampled quality gateEach cluster samples its own outputs before any model version is promoted there.
- 5Audit recordRouting metadata is logged centrally, without any customer content.
Actions taken
- Adopted a regional split for source text and inference, with a single control plane that holds no customer content and routes each request to its jurisdiction's cluster.
- Batched long documents into their own length-bucketed queue, so a single large filing could no longer delay shorter jobs, and set a separate p99 alert for each bucket.
- Extended the quality gate so each regional cluster samples its own outputs, with reviewers drawn from the same language pool, and blocks promotion per region when a threshold fails.
- Set a standing model upgrade window with a regional rollout order, and kept the tokenizer and runtime pinned together so no instance could drift from its peers.
Results
The regional split went live in stages over the following weeks, starting with the largest contract's jurisdiction. p99 latency on long documents dropped below the ten-second ceiling in the first full week, mostly from the length buckets. Monthly inference spend rose by about 18 percent instead of the 40 percent the team had projected for a full second region, and both residency customers accepted the design after their security reviews. The quality gate caught one regional threshold failure during the first upgrade, which blocked promotion for a single cluster for two days and was the first real test of the design.
Questions for discussion
- Which of the three options would you have rejected earliest, and which evidence would have to change your mind?
- How should a team weigh an operating cost that grows with each new jurisdiction against a single fixed design?
- What would you need to measure in the first month to know whether length bucketing was the real fix?
Case Study 5
Evaluating an automated resume screen before any hiring decision
Pelham Recruiting Partners, a staffing firm that places clerical and technical workers for about 60 client employers.
The situation
Pelham Recruiting Partners receives roughly 3,400 applications a month across its placement desks, and its recruiters spend most of their time reading resumes that never progress. A vendor offered a screening model that ranks applicants against each job's requirements and proposes a shortlist of the top 15 percent. The vendor's pitch promised a steep cut in screening hours. The operations director, Mirela, was skeptical of a single vendor summary and asked for a pilot on the firm's own past placements before any recruiter relied on the tool. The pilot had to run alongside existing work without changing who was contacted.
The challenge
The tool had been trained on the vendor's pooled data from other staffing firms, not on Pelham's own placements. Early reviews suggested it weighted years of experience heavily and treated employment gaps as a negative signal, which could penalize people returning from caregiving or illness. Pelham's recruiters also suspected that the model favored applicants whose resumes followed a conventional layout, since parsing errors were common on two-column templates. The team needed to know whether the efficiency gain would hold without quietly narrowing who got interviews.
Analysis
Begin with what the screen is allowed to do. In this setting the tool only proposes; a recruiter still decides who is contacted, and the firm's client contracts require that every rejection be explainable to the applicant on request. Those two constraints rule out any design in which the model's ranking is the final step. The operating data for the pilot came from three sources: the model's scores, the recruiter's decision, and the eventual placement outcome for the hires made over the prior several months. Comparing the three showed where the tool and past practice disagreed, which mattered most. Scores alone would have said little about fairness.
Tolerances were set before the pilot started. The team agreed that the shortlist could differ from recruiters' choices by no more than 12 percent of names, that the score gap between shortlisted groups should stay within 5 points of the recruiter baseline, and that any unexplained rejection rate increase above 2 points would stop the pilot. Maintenance records supplied the hardest part. The vendor's parser had been updated twice recently, and the changelog did not mention the template handling change that affected two-column resumes. Recording each model version, parser version, and screening threshold in the pilot log made it possible to separate a tool problem from a data problem.
- Before any scoringTolerances, rollback rule, and sign-off owners written and agreed.
- Start of pilotScores logged for every application while recruiters worked as before.
- Second placement cycleScore gap crossed the 5 point limit, and the parser change was traced.
- After the vendor fixGap held within limit for a full cycle before senior roles were reconsidered.
Actions taken
- Ran the screen in shadow mode for the full pilot period, so scores were logged but never shown to recruiters or used to contact applicants.
- Broke results out by applicant group, resume layout, and employment gap length, and reviewed each slice with a recruiter before any change to the threshold was approved.
- Required a named recruiter to sign off each shortlist and to record a reason whenever they overrode the model, with overrides reviewed weekly by operations.
- Set a rollback rule in the pilot runbook: any tolerance breach for two consecutive weeks returns the screen to shadow mode, with the vendor notified in writing.
Results
The pilot ran through the next two placement cycles. The shortlist differed from recruiter choices by about 9 percent of names, inside tolerance, but the score gap between groups widened in the second cycle and crossed the 5 point limit. The team traced the widening to the parser change, not to the model's weights, and the vendor issued a fix. Recruiters valued the time saved on high-volume roles, but the firm kept the tool off senior placements until the gap held for a full cycle. Screening hours fell by about a fifth where the tool was used.
Questions for discussion
- Which tolerance in this pilot would you have been most tempted to relax once the efficiency numbers arrived, and why?
- How much weight should a recruiter's override reasons carry when the model is later tested against those same overrides?
- What would convince you that a fairness gap came from data handling rather than from the model's learned weights?
Learning Resources
Technical CASE STUDIES
China, India, United States, Indonesia, Pakistan, Nigeria, Brazil, Bangladesh, Russia, Japan, Mexico, Philippines, Vietnam, Egypt, Turkey, Germany, United Kingdom, France, Italy, Canada, South Korea, Australia, Spain.