Generative AI Applications Case Studies
Technical5 Case StudiesUpdated Sep 2026
Five case studies follow, each set inside a fictional organization that builds or runs tools on large language models. They are written for practitioners who have to decide what reaches production, how its performance is shown, and who may touch it once it is live. Every case is read through the same four lenses: system requirements, performance evidence, access controls, and maintainability. The scenarios move from choosing a single growth bet, to diagnosing a recurring failure, to settling a dispute between teams, to reading metrics honestly, and finally to weighing a shortcut that carries long-term exposure. Each case closes with discussion questions for a small group.
Case Study 1
Choosing one growth bet with a four person engineering team
Harbor Lane Logistics, a fictional regional freight brokerage with about 140 staff and a four-person platform team
The situation
Harbor Lane Logistics runs freight brokerage for about 140 staff and keeps a platform team of four engineers. Leadership wanted one new feature from the language model work they had prototyped over the last two months, and the budget allowed roughly one build cycle before the next planning review. Three candidates were on the table. Dispatchers wanted a summary of shipment exceptions written in plain language, a need raised in two planning sessions. Sales wanted a draft generator for carrier quotes, which reps said they would use daily if it worked. Compliance wanted a classifier for customs documents, which had the highest volume of manual effort but touched regulated personal data.
The challenge
The team could only finish one of these well. Each candidate had a different model of success. The summary feature would be judged by dispatchers reading a few sentences, so latency and accuracy on exception codes mattered most. The quote generator needed retrieval against rate tables that changed weekly, which pushed up maintenance. The classifier would require strict controls on who could see document contents and where logs were stored. Leadership wanted a recommendation with evidence, not enthusiasm, by the end of the week.
Analysis
Teams grounded in Generative AI Applications usually start by writing down what the feature must do before comparing vendors or model sizes. For the dispatcher summary, the requirement was narrow: produce a three-sentence description of an exception within a few seconds, and never invent a carrier name or timestamp. That narrowness made the evidence easier to gather. The team pulled 60 past exception records, had two dispatchers grade the drafts blind, and measured how often a draft held a fact missing from the source. The quote generator had no comparable test set, and the sales demo looked good only because the reps chose the examples.
Access and upkeep should shape the choice as much as the demo. The classifier touched personal data in customs filings, so the team would need role-based access to document text, redaction before any logs were written, and a documented retention period agreed with compliance. Those controls are cheap to design early and expensive to retrofit. The quote generator, by contrast, depended on rate tables maintained by two people outside engineering, so any feature built on them inherited a weekly update burden. The team recommended the dispatcher summary, which had clear evidence, narrow access needs, and a single owner for its prompt and test set.
Actions taken
- Write a one page requirement for the dispatcher summary that names the three fields it must never alter, and get sign off from the dispatch lead before any build work starts.
- Build a fixed test set of 60 exception records with graded reference notes, and rerun it on every prompt change so regressions show up before dispatchers see them.
- Restrict the summary service to the dispatch role, log only request identifiers rather than shipment text, and set a seven day retention window on the cached drafts.
- Park the quote generator until rate tables have a named owner and an update process that does not depend on a single spreadsheet, then revisit it at the next planning review.
Results
The summary went to dispatchers in the third week after the decision. Across the first two weeks, reviewers accepted 71 percent of drafts with no edits, and the blind test found a missing-fact error in 3 of the 60 records, down from 9 in the first prompt version. Average generation time stayed under four seconds on the internal hosting setup. The quote generator stayed on the backlog. Sales asked twice for a timeline, and the answer was tied to naming an owner for the rate tables. The classifier was scoped for later with compliance involved from the start.
Questions for discussion
- Which evidence would have changed the ranking if the sales demo had been run on randomly chosen accounts?
- Which of the four controls would you insist on before any feature touches customs documents, and why?
- How would you decide when a deferred feature deserves a second look without relying on enthusiasm?
Case Study 2
Recurring handoff failures in a patient messaging assistant
Bellmont Health Records, a fictional scheduling software vendor serving about 60 outpatient practices
The situation
Bellmont Health Records sells scheduling and messaging software to about 60 outpatient practices. Two months earlier it added a language model assistant that answers routine patient questions and hands anything unresolved to a human coordinator. Within a month the handoff queue grew from a steady 20 tickets a day to more than 90 on busy days. Practices reported that patients repeated themselves after transfer, and the share of chats resolved without staff fell from 64 percent to 48 percent. The support group of 45 was absorbing the spike while the engineering team argued about prompts.
The challenge
The obvious suspect was the model itself, since a vendor update had landed a week before the spike. The team needed to separate that from intent detection, the routing rules that decide when to hand off, and the summary it passes to the coordinator. Each of those could fail in a different way, and each had a different owner. The response had to lower handoff volume without cutting off patients who needed a person, and it had to show results within the next reporting cycle.
Analysis
Before changing anything, the team mapped the path a message took from first contact to resolution, with a timestamp and an owner at each step. That map showed the handoff step was the only place where the model's output became a human's input, so the summary text was a system requirement in its own right, not a courtesy. The team then tested causes one at a time against a set of 400 archived conversations that had produced handoffs. They replayed each conversation with the old model version, the new version, and the original routing rules, and scored whether the summary held the patient's stated reason for contact.
The replay also exposed an access problem. Coordinators were seeing full transcript text in the handoff queue, including content from patients who had not yet been verified, and the routing service had broad read rights that nobody had reviewed since launch. The fix had two parts: limit transcript visibility until identity is confirmed, and separate the routing configuration from the prompt so that a coordinator-facing change could not silently alter routing. Keeping those pieces apart also made the system easier to maintain, because the routing thresholds could be tuned by the operations lead without touching prompt text that the vendor's model team owned.
- Queue builds Routing sends unresolved chats to staff before summaries are complete.
- Patients repeat Patients repeat their reason because the summary omits it.
- Staff slow resolution Coordinators absorb more chats, which lengthens every queue wait.
- Thresholds tuned blindly Routing changes ship without stage data, so the loop repeats after each release.
Actions taken
- Add a timestamped stage field to every conversation record so that handoff delay can be measured separately from model latency, and publish the first breakdown within two weeks.
- Restrict transcript visibility in the coordinator queue to a summary field until identity verification is complete, and log each full-transcript view with a reason code.
- Split routing thresholds into their own versioned configuration file, owned by operations, with a rollback path that takes under ten minutes.
- Set a target that handoffs stay below 35 percent of conversations and that repeated-question complaints fall by half over the next reporting cycle, and review both weekly.
Results
Within the reporting cycle, the stage breakdown showed that 70 percent of handoff delay came from the queue, not from model response time, which redirected effort toward staffing and visibility rules. Handoffs fell from more than 90 on peak days to 52, and resolution without staff returned to 61 percent. Repeated-question complaints dropped from 14 in a week to 6. The vendor's model update was reviewed separately and found to have a small effect on summary accuracy, which was corrected in the next release. One practice reported slow pickup on weekends, a staffing question the operations lead took to the customer.
Questions for discussion
- Which of the four measurements would you collect first if the stage field could not be added for another month?
- How would you decide whether a visibility restriction is protecting patients or hiding evidence the team needs?
- What would make you distrust a replay result that shows the new model version scoring better on summaries?
Case Study 3
Negotiating a claims assistant between operations and compliance
Quillfeather Claims Group, a fictional claims administrator with roughly 900 staff across four service centers
The situation
Quillfeather Claims Group administers property and auto claims for regional insurers and employs roughly 900 people across four service centers. A product team had built a prototype that drafts claim status letters and first-pass coverage explanations from the adjuster's notes. Operations wanted it live within the quarter because average handling time had drifted up, and the service center directors had committed to a faster turnaround for two insurer clients. Compliance had not signed off on any text leaving the building without a named human reviewer. Security was reviewing whether the model could run on the company's own cloud account or had to go through a vendor's hosted endpoint.
The challenge
Three leaders held decision rights over different parts of the rollout, and none of them had been asked to agree on the whole. The operations vice president could approve a pilot but not a change to customer letters. The compliance officer could block release but had no budget for staffing review. The CIO controlled hosting but reported the cost of a dedicated deployment in a separate budget. The product lead needed a yes or no on a pilot before the next planning session, and each group framed it differently.
Analysis
The clearest way through was to translate each group's concern into a requirement that could be tested. Compliance's concern became a rule that no letter leaves the building without a logged human edit and a named reviewer. Operations' concern became a measure: handling time on a sample of real files, compared with the same adjusters working unassisted. Security's concern became a boundary: prompts, claim data, and drafts stay inside the company's own cloud account. The prototype evidence was thin. It had been scored on 30 letters the product team wrote, so the team proposed a pilot on 200 closed files, with two senior adjusters grading drafts blind.
Access and upkeep turned out to be where the agreement could hold or fail. Decision rights needed to be written down: operations approved pilot scope, compliance could halt any release lacking logged review, and the CIO owned the hosting boundary. Access followed the same logic. Only adjusters on assigned files could generate drafts, and reviewer edits had to be retained for the audit period compliance specified. For maintainability, prompt changes went through the same review as letter templates, so no wording change reached customers unseen by compliance. The workable agreement was a one-page charter naming an owner for each decision and a review date.
Actions taken
- Hold a single working session with the three leaders and write the pilot scope as testable statements, one for each concern, and get each leader to initial it.
- Run a blind pilot on 200 closed files, with two senior adjusters scoring each draft for factual accuracy, tone, and missing required disclosures against a fixed rubric.
- Enforce the hosting boundary in the cloud account from day one, with prompts, claim data, and drafts kept in a single approved region and no outbound calls to the vendor endpoint.
- Set a joint review at the end of the pilot where operations, compliance, and the CIO each decide go, extend, or stop against the agreed measures, with the charter updated in writing.
Results
The pilot ran for one quarter. Blind grading found that 82 percent of drafts needed no factual correction, and handling time on the sample fell by 19 percent against unassisted adjusters. Compliance accepted the pilot once reviewer edits were logged in every case, and it found two template phrases that required disclosures the model had omitted, which were fixed before expansion. The CIO kept hosting inside the existing account, which avoided a separate deployment budget. Operations extended the pilot to one service center only, and the agreement stayed unchanged for the other three until the next review.
Questions for discussion
- Which of the three concerns would you have been least willing to trade away, and what evidence would have changed that view?
- How would you handle a case where one leader holds the decision right but another holds the budget that determines whether it can happen?
- What should the charter say about who may stop the pilot mid-run, and who must be consulted first?
Case Study 4
Strong usage numbers hide a missed repair target
Northfield Utility Services, a fictional maintenance contractor with about 2,300 employees across three client regions
The situation
Northfield Utility Services maintains distribution equipment for three regional clients and employs about 2,300 people, roughly 700 of them in field crews. Six months earlier the company rolled out a retrieval assistant that lets technicians ask questions about equipment manuals, past work orders, and safety procedures from a phone. The dashboard reported strong activity: 4,800 queries in the first month, and more than 80 percent of technicians had used it at least once. The operations committee had set a target that first-time fix rate would rise from 71 percent to 78 percent by the end of the second quarter. At the most recent review, the fix rate had moved by less than one point.
The challenge
Activity was not the problem. Usage looked healthy in every report the committee received, which made it harder to see why the outcome was flat. The team needed to find out whether the metric itself was measuring the wrong thing, whether the assistant was giving answers that technicians could not act on, or whether the target assumed a change in the field that the tool could not cause. Each explanation pointed to a different fix, and the next review was in three weeks.
Analysis
The first step was to check whether the indicators could support a decision at all. Query volume counted every search, including repeated searches for the same safety procedure and tests run by the training lead, so it said little about whether a repair got done. Active users counted anyone who opened the app once. Neither figure had a defined requirement behind it. The fix rate was the right outcome, but its definition had drifted: crews recorded a repeat visit as a fix when the first one was logged as incomplete, and two of the three clients counted fixes on different calendars. Nothing recorded whether the advice was followed.
Access rules also affected what could be measured. Field crews saw only their own work orders, so joining assistant logs to client records required a shared service account with broad read rights. Tightening it would make the numbers harder to pull, which is the right trade, but it had to be planned. For maintainability, the team proposed a metric dictionary with one owner per indicator, a change log for definitions, and a monthly review that compares each indicator against an outcome it is supposed to predict. The aim was a cycle that the operations analyst could run without rebuilding dashboards each time someone asked a new question.
Actions taken
- Rewrite the fix rate definition so that a repeat visit counts as a failure of the first visit, and publish the change log to all three clients before the next review.
- Add an outcome link to every assistant session by recording the work order identifier and the technician's close code, without storing the question text in the reporting layer.
- Replace the shared service account with two scoped read roles, one for the operations analyst and one for the dashboard, each reviewed every month by the security lead.
- Hold a monthly metric review that asks whether each indicator moved with fix rate over the previous three months, and retire any indicator that cannot answer that question.
Results
The first corrected monthly report showed the fix rate at 73 percent, a real move but well short of the 78 percent target. Query volume dropped by about a third once repeat searches were removed from the count, which made the drop look worse on paper but more honest. The linked close codes showed that 62 percent of assistant sessions ended in a completed visit, against an assumed figure of 80 percent. The committee agreed to keep the target and to fund a review of the parts procurement process, which turned out to be the bottleneck in the field.
Questions for discussion
- Which indicator on the dashboard would you retire first, and what would you need to see before you were willing to remove it?
- How would you explain to a client why a lower query count can represent better reporting?
- What would it take for the team to conclude that the target itself was wrong rather than the assistant?
Case Study 5
Reusing client transcripts to improve a shared model
Larkspur Software, a fictional developer of field service scheduling tools with about 210 employees
The situation
Larkspur Software builds scheduling software for field service companies and employs about 210 people. Its assistant drafts dispatch messages and answers technician questions, and it runs on a hosted language model that the company tunes with its own examples. Over the last quarter the tuning backlog grew, and the team found that transcripts from 40 client accounts, if pooled, would double the number of usable examples for one task. Sales had promised two clients faster accuracy gains. Most contracts allowed use of usage data to improve services, but the wording was broad, and three of the largest clients had asked in writing that their data stay separate.
The challenge
The decision was whether to pool the transcripts into the shared tuning set, keep each client's data in its own tuning set, or use only the synthetic examples the team could generate. Pooling gave the fastest gain and the lowest cost. Keeping data separate was slower and more expensive, and the synthetic route might not reach the accuracy the sales team had promised. The company's general counsel wanted a recommendation that could be explained to clients without qualification.
Analysis
The performance case for pooling was real but narrow. The team's offline test showed a gain of roughly six points on the dispatch task when the pooled set was used, but almost all of that gain came from three large accounts whose tickets used similar vocabulary. Gains for smaller clients were within the noise of the evaluation. A system requirement that the tuning pipeline could identify the source account of every training example, and remove it on request, turned out to be absent from the current build. Without it, a deletion request could not be honored without retraining from scratch, and the gain could not be separated by source.
The exposure grew in access and upkeep. A pooled model would inherit every client's transcripts in its weights, so removing one client later means retraining, not deleting rows. Access to the shared tuning set would also widen from the four engineers who now handle client data to every team that uses the model. The consequences reach beyond the company, the three clients that asked for separation would have grounds to terminate and their contract reviews would surface it quickly. A safer path keeps per-client tuning sets, adds source tagging and deletion, and offers the faster gain only to clients who opt in.
- 1Pool transcripts nowFastest gain, but deletion and separation promises become unenforceable.
- 2Tag every exampleEach training item carries its source account for audit and removal.
- 3Build deletion workflowRemoving a client triggers retraining within a stated window.
- 4Opt-in clauses for poolingClients who agree get shared tuning with clear terms and withdrawal rights.
- 5Separate sets for othersRemaining clients keep isolated tuning, which costs more but holds trust.
Actions taken
- Freeze the pooled tuning job and document the current data sources, so that no transcripts from the three separation-requesting clients enter any shared set while the review runs.
- Add a source account tag to every training example and a deletion workflow that removes a client's examples and triggers retraining within a stated window.
- Offer pooled tuning only to clients who opt in through a separate clause that names the data use, the retention period, and the right to withdraw.
- Give sales a revised accuracy forecast for each client tier based on per-client tuning results, and brief the two promised clients before any further commitment is made.
Results
The company kept per-client tuning sets and shipped the source tagging. Opt-in clauses were accepted by 9 of the 40 accounts in the first round, and the two clients who had been promised faster gains received a revised forecast that was smaller than the original promise. Per-client tuning recovered about half of the pooled gain on the dispatch task. Sales lost one expansion deal that depended on the earlier promise, and the company judged that the loss was the cost of keeping the three largest accounts. Legal reviewed the deletion workflow and asked for a test.
Questions for discussion
- How would you weigh the sales commitment against the risk to the three largest accounts if the decision had to be made within the week?
- What evidence would convince you that a deletion workflow is trustworthy before it is promised to a client?
- Which safeguard in the opt-in clause would you test first, and how would you show that clients understood it?
Learning Resources
Technical CASE STUDIES
China, India, United States, Indonesia, Pakistan, Nigeria, Brazil, Bangladesh, Russia, Japan, Mexico, Philippines, Vietnam, Egypt, Turkey, Germany, United Kingdom, France, Italy, Canada, South Korea, Australia, Spain.