Natural Language Processing Case Studies
Technical5 Case StudiesUpdated Sep 2026
Language systems rarely fail where people look first. The model gets the attention, while the normalizer, the label guide, the retention policy and the on-call rota decide how often things break and how quickly they recover. The five cases here come from teams running or evaluating text-based systems for real users. Each is organized around the same four concerns: what the business needed, what evidence existed, which operational limits shaped the options, and who carried the consequences. Read them as working notes rather than verdicts, since the details carry most of the lesson.
Case Study 1
Routing regression after a release on a busy week
Harbor Lane Support Systems, a regional software vendor whose ticket routing service handles requests for about 340 customer support teams.
The situation
Harbor Lane runs a ticket routing service that reads incoming support messages in six languages and sends each one to the right team. Three days after a model update and a preprocessing change shipped together, the general queue began filling faster than staff could clear it. Within the first hour, the share of tickets falling to the fallback route rose from about 6 percent to more than 30 percent. Customers in two markets reported that replies were going unanswered for most of a day. The on-call engineer, Marisol, found that the release had bundled a new text normalizer with a retrained model, and the two changes had never been tested together against the non-English slices.
The challenge
The business needed routing restored before the weekend coverage gap, since two enterprise accounts had service credits written into their contracts for delayed first responses. Engineering could roll back the whole release in under half an hour, but that would also discard a spam filter fix the trust team had been waiting on. Logs were sampled at one in ten for cost reasons, so the evidence was thinner than anyone wanted. Nobody could say yet whether the regression lived in the normalizer, the model, or the interaction between them.
Analysis
Start with what the service was for. Harbor Lane did not need a perfect classifier during the incident. It needed tickets reaching people inside the contracted window, and that reframes the first decision: stop the damage before explaining it. Experienced responders separate the question of what broke from the question of how to restore service, because the evidence that answers the first usually arrives too late for the second. Here the sampled logs still showed the normalizer altering accented characters and some non-Latin punctuation before tokenization, which made the new model look worse on exactly the slices that had been stable before.
The useful evidence came from comparing the two model versions on a frozen sample of 2,400 tickets from the prior week, sliced by language and by whether the text contained diacritics. The regression concentrated in exactly those slices. That pointed away from the model weights and toward preprocessing, which was cheaper to change. The team kept the retrained model, reverted the normalizer through a feature flag that had been added for an unrelated reason, and scheduled a shadow run before reintroducing it. Stakeholder impact was measured in the queue rather than on the dashboard: support leads counted how many customers had waited more than a full business day, and that count drove the service credit calculation.
Actions taken
- Reverted the normalizer behind its existing flag within the first hour, leaving the retrained model live, and posted the change to the support leads so they could reprioritize the general queue.
- Replayed the failing slices offline against the previous artifact and the candidate pair, so the fix would be judged on the same sample that exposed the regression rather than on a fresh guess.
- Added a release gate that required per-language and diacritic slice scores to stay within a fixed margin of the baseline, with the gate written into the deployment checklist rather than kept in someone's notebook.
- Raised log sampling for the routing path to full capture for the first two weeks after any preprocessing change, accepting the storage cost because the sampled logs had hidden the problem for a day.
Results
Service returned to its usual fallback rate of under 7 percent by the end of the afternoon. The two accounts with credit clauses received the credit for delayed responses, calculated from the waiting counts and reviewed line by line by finance. The normalizer fix shipped nine days later, after the replay showed the accented-character slices back within tolerance of baseline. The release gate caught a similar drift in the next model update, which was held and corrected before it reached any customer. The incident review concluded that the sampling policy was the real gap, more than the code.
Questions for discussion
- Which slice would you have added to the release gate first, and what is the smallest sample you would trust for it?
- When a rollback would also remove a fix that someone else is waiting on, how would you choose between a partial revert and a forward patch?
- How should contractual penalties such as service credits shape the order of recovery steps, if at all?
Case Study 2
Conflicting dashboard totals traced to a null handling change
Tallgrass Mutual, a regional property insurer whose claims adjusters file about 1,200 free-text notes in a typical period.
The situation
Tallgrass Mutual extracts cause-of-loss categories from free-text adjuster notes with a text classification model, then writes the labels into a warehouse table that three reporting teams read. In the second month of the quarter, the claims operations director noticed that the water damage share on the regional dashboard was about 4 points higher than the figure in the actuarial pack. Both numbers claimed to come from the same extraction. Adjusters had filed roughly 1,200 notes in the period, and the gap had gone unnoticed because each team checked only its own totals against the prior quarter. No one had changed the model in that window, but a schema migration had altered how empty notes were stored.
The challenge
The decision at stake was a proposed change to regional reserve assumptions, which leadership wanted to base on the higher share. Actuarial wanted the existing basis kept until the discrepancy was explained. Leadership had a review meeting in the third week, and each team was working from a different extract of the same notes. The warehouse held no lineage record for the label column, and the notes were partly redacted for privacy, which limited what anyone could inspect directly.
Analysis
The business objective was narrower than fixing the pipeline. Leadership needed to know whether the reserve change was justified, which meant deciding which number was defensible for that decision, not which number was correct in the abstract. Teams grounded in Natural Language Processing often frame reliability this way, asking what error rate the decision can tolerate before asking how to drive error toward zero. In this case the evidence sat in three places: the raw notes in the claims system, the extraction job logs, and the warehouse load history. Only the first two were trusted by the people who owned them, so the trace had to run through all three.
Operational constraints shaped the validation plan. Re-running the full classifier over the period was possible but would take most of a week, because the batch queue was shared with underwriting. Redacted notes could be read only inside the secure environment, and anything leaving it had to be aggregate counts. The team therefore drew a stratified sample of 300 notes, had two adjusters label them without seeing the model output, and compared agreement by region and by the presence of an empty field. Stakeholder impact was lopsided. An inflated water damage share pushed reserves up, an undercount pushed them down, and the adjusters whose notes fed the labels had no view of how those labels were used.
- 1Adjuster note entryBlank notes were saved as empty strings before the schema change.
- 2Schema migrationBlank strings converted to nulls with no notice to consumers.
- 3Extraction stepNulls passed through as empty text and received the fallback label.
- 4Warehouse loadRegional view counted fallback rows, actuarial view excluded them.
- 5Dashboard refreshTwo reports disagreed by about 4 points without raising any error.
Actions taken
- Traced the label column back through each transformation, recording the query, the join keys, and the default applied when a note was empty, so every dashboard number could be reproduced from one documented path.
- Found that a schema migration had turned blank notes into nulls, which passed through as empty text, so the fallback label was applied and only one view counted those rows.
- Added row count and label share checks at each load, with thresholds drawn from the prior four quarters and alerts sent to the warehouse owner.
- Re-labeled the stratified sample of 300 notes with the corrected pipeline and reported agreement with the adjusters' reading alongside the old and new totals, so leadership could see the size of the change.
Results
The review went ahead with the corrected figure, which sat closer to the actuarial pack but still about 1 point above its basis once empty notes were handled explicitly. Leadership deferred the reserve change for one quarter and asked for a documented basis instead. Two months later the load checks caught a smaller shift caused by a vendor template change, and it was fixed before any report refreshed. Reporting teams now point to a single lineage document. Adjusters received a short summary of how their notes became labels, which cut repeat questions to the claims desk.
Questions for discussion
- How would you choose between two inconsistent figures when neither is fully verified and a decision is due next week?
- What minimum lineage record should a label column carry before any team is allowed to report from it?
- Who should own the alerts on a shared warehouse table, and what should they be expected to do when one fires?
Case Study 3
Suspicious API activity during a filing deadline
Gilcrest Legal Technologies, a firm that hosts contract clause extraction for mid-sized law practices and handles privileged client documents.
The situation
Gilcrest runs a hosted clause extraction service that law practices use to review contracts. Each tenant authenticates with its own API keys, and every request is logged with the input text, which is itself privileged client material. Early one evening, the monitoring team noticed one tenant's key making requests at a steady rate from an address range that tenant had never used. Over two hours, about 4,000 requests had retrieved long documents from storage. The on-call analyst, Teodor, could not tell whether the tenant's staff were working late or whether the key had leaked.
The challenge
The business concern was twofold. Several tenants were in active litigation, and disclosure of privileged text would be a serious professional and contractual failure. Suspending the tenant would also stop a team in the middle of a filing deadline. The request logs were the main evidence, but they were kept on a fixed retention schedule, and the forensics vendor needed the tenant's authorization before touching its environment. The tenant's administrator would not be reachable until the next morning.
Analysis
The first task was to separate what the evidence can establish from what it cannot. The request logs show which key was used, from where, against which document identifiers, and at what pace, but they do not show who was at the keyboard. That gap matters because the containment options differ. Revoking a key is cheap and reversible, while suspending a tenant is neither. Responders usually treat the prompt and retrieval logs as the core evidence here, since they show what text actually reached the model rather than only which endpoint was called. The team preserved those logs in a write-once store before any cleanup job could run.
The constraints were mostly procedural. The retention policy purged request bodies after a fixed period, so the team exported the relevant window first and recorded a hash of each file. Authorization for the vendor came through the tenant's named administrator, and the runbook required that every decision to revoke, rotate, or suspend be recorded with its reason and approver. Stakeholder impact fell on three groups: the affected tenant, the other tenants sharing the inference cluster, who would notice any throttling, and in-house counsel, who had to judge whether a notification duty existed at all.
- Evening alertSingle key showed steady requests from an unused address range.
- Two hours laterAbout 4,000 requests had retrieved long documents from storage.
- Revocation stepOnly the suspect key was revoked, through the tenant administrator.
- Same dayLogs exported to write-once storage and hashes verified.
- Next morningCounsel and tenant reviewed scope before any notification decision.
Actions taken
- Revoked only the suspect key and issued a new one through the tenant administrator channel, leaving the tenant's other users and integrations untouched, and logged the revocation time and approver.
- Exported the request and retrieval logs for the affected window to a write-once store, computed hashes for each file, and kept the original retention job paused until counsel confirmed the copy was complete.
- Scoped the incident by listing every distinct document identifier retrieved under the key, then checking whether any belonged to matters outside the tenant's permitted set, which narrowed the review to 41 documents.
- Documented each step with its authority, time, and reversibility, and set a review with counsel and the tenant for the next morning before any suspension or notification decision.
Results
Revocation stopped the traffic within minutes of the decision. The export finished before the retention job would have purged the window, and the hashes matched on verification. The scoped review found 41 documents retrieved, all within the tenant's permitted matters, which was consistent with a leaked credential in a shared script but did not prove it. Counsel concluded that no notification duty had been triggered and recorded the reasoning in full. The tenant later found the script in a contractor's repository, and the other tenants on the cluster saw no interruption.
Questions for discussion
- What evidence would have justified suspending the whole tenant rather than revoking a single key?
- How should a team weigh speed of containment against the risk of losing logs that a later review will need?
- Which decisions in an incident like this should require a named approver, and which can safely stay with the on-call analyst?
Case Study 4
Choosing where semantic search computes its embeddings
Thornbury Index Systems, a small vendor that provides semantic search over archived correspondence for municipal records offices.
The situation
Thornbury sells semantic search over archived letters and memoranda to municipal records offices. The combined index holds 2,300,000 documents. Over the past several quarters the product grew from three pilot customers to nineteen, and the prototype that won the pilot was never built for that load. Every query called a hosted embedding API, then passed the top 200 lexical hits through a cross-encoder reranker. At current volume the 95th percentile response time sat near 2.4 seconds, and the hosted API bill was rising faster than the contracts that paid for it. Two customers also raised residency questions, since their archives contain personal information that cannot leave their jurisdiction.
The challenge
The goals pulled in different directions. Records staff wanted results under a second at the 95th percentile, the finance lead wanted infrastructure cost held below a fixed share of revenue, and the two residency customers would not sign renewals without local processing. The team had four engineers, one of whom was on rotation for the hosted services, and nobody had run GPU capacity in production before. Each option promised something and quietly moved a different kind of risk onto the team.
Analysis
The first question was what the search had to get right. Records staff judged results by whether a known letter appeared near the top, not by a model score, so the team built a labeled set of 1,100 query and document judgments from staff who had handled those requests. On that set lexical retrieval reached recall at ten of 0.71, dense embeddings reached 0.84, and dense plus reranking reached 0.87. The reranker's gain was real but small next to the latency it added. Building that benchmark came first, since an architecture debate without a labeled judgment set tends to end in preference rather than evidence.
Constraints and stakeholders then narrowed the field. Residency rules ruled out fully hosted embedding for two customers, so those tenants needed self-hosted capacity, which meant operating and patching GPUs with a team that had no rota for them. The reranker's cost scaled with candidate count, which made the 200 candidate setting a budgeting decision as much as a quality one. Records officers also needed to understand why a result ranked where it did, which favored a visible lexical stage over a single opaque dense score. Reliability cut the other way too: a hosted outage had already silenced two customers for most of an afternoon.
Actions taken
- Ran a two-week shadow comparison of three candidate pipelines on replayed production queries, measuring recall, p95 latency, and cost per thousand queries, so the decision used observed traffic rather than vendor estimates.
- Moved embedding generation for the archive to self-hosted capacity inside the municipal network for the two residency customers, keeping hosted calls only for tenants without residency requirements.
- Kept lexical retrieval as the first stage and limited the reranker to the top 50 candidates with a timeout that fell back to first-stage order, which bounded tail latency during reranker load.
- Set up a monthly review of recall on the labeled set and a cost dashboard with budget alerts, and assigned ownership to the platform lead rather than leaving it with the search team.
Results
Latency for the two residency tenants settled near 900 milliseconds at the 95th percentile, and the hosted tenants stayed under 700 milliseconds, both inside the one-second target once the reranker was capped and timed out. Recall at ten held at 0.85 on the labeled set, close to the full pipeline and well above lexical alone. Monthly infrastructure spend fell to a level the finance lead accepted, though the self-hosted tier needed a second engineer on call within the first quarter. Both residency customers signed renewals in the next cycle. Records staff said the visible lexical stage made disputed rankings easier to explain, which they valued more than the last few points of recall.
Questions for discussion
- What labeled evidence would you need before accepting a small recall gain in exchange for a large latency cost?
- When a residency requirement forces self-hosting, what operational commitments should be agreed before a contract is signed?
- How would you keep the cost model honest if query volume grows faster than the forecast?
Case Study 5
Evaluating a claims triage model before it touches members
Oakhollow Benefits Administration, a third-party administrator that processes employer health plan claims for mid-sized firms.
The situation
Oakhollow processes roughly 60,000 health plan claims a month for employers that outsource benefits administration. Its claims team of 45 adjusters handles routine claims by hand, with a median turnaround of about two days. A vendor proposed a language model that reads claim narratives and attached provider notes, assigns each claim a routine or review category, and drafts explanation letters for routine approvals and partial denials. The vendor's pilot showed a large drop in handling time on a sample of 2,000 claims, and the operations director called it the strongest efficiency case she had seen from any vendor.
The challenge
The pilot did not measure what the business needed. Handling time was the headline, but the errors that mattered were wrongful denials and inconsistent treatment of claimants, and those were rare enough that a sample of 2,000 claims could not separate the approaches. Two employer clients had already asked how a machine decision would be explained on appeal. Written denial rules are strict, and a letter citing the wrong reason creates a compliance problem as well as a service complaint.
Analysis
The vendor sold one outcome, and the operations director was right to want it, but the evaluation has to measure two. Handling time is visible on her dashboard. Wrongful denial is a harm to a specific member, usually discovered only on appeal, and it does not average out across a population. Both must be measured on the same claims, with each error weighted for what it costs the person affected. Evaluators often build the error taxonomy first, listing the mistakes a person could make and what each one costs, because that list determines how large a sample must be before anyone can call a rate low.
The constraints were severe. Member data could not go to an external model without a data processing agreement that would take weeks to negotiate, so the pilot ran in a restricted environment on a fixed budget. Adjuster time was the scarce resource: re-reviewing a stratified sample of 400 claims each week consumed nearly a full adjuster's week, which the team accepted as the price of oversight. Stakeholders wanted different things. Employers wanted faster decisions, members wanted correct decisions and readable letters, and adjusters feared that reviewing machine output would become a rubber stamp. Those pressures explained why the team slowed down when employers started asking about appeals.
- Measure Track error rates by claimant group, not only overall accuracy.
- Sample Pull stratified claims for adjuster re-review every week.
- Adjust Tighten thresholds or retrain when a group drifts out of range.
- Pause Return affected categories to manual queues until the gap closes.
Actions taken
- Built a held-out evaluation set of 3,000 claims, stratified by claim type, claimant language, and provider category, with each claim independently coded by two adjusters before any model output was shown to them.
- Set pass criteria in advance: no group's wrongful denial rate could exceed the overall human baseline by more than a fixed margin, and the model could not auto-deny anything in the first phase.
- Rolled out to routine approvals on a single employer's claims with one in twenty auto-approvals pulled for random review and a one-click reversal available to adjusters for seven days.
- Gave adjusters the authority to stop the rollout in any category, wrote a plain-language explanation template that a member could act on, and reviewed appeal outcomes monthly against the model's stated reasons.
Results
The evaluation showed the model matching the two-adjuster reference on routine approvals at an acceptable overall rate, but worse on claims with non-English narratives, where wrongful denials ran roughly double the human baseline. The team did not open denials to automation at all. Auto-approval went live only for the categories that passed, and handling time in those categories fell by about a third. The random review pulled two approvals that adjusters would have sent for a second look, and both led to a template change. The employer clients received the criteria in writing, and one of them asked to be included in the next phase.
Questions for discussion
- What wrongful denial rate would you treat as unacceptable for a single claimant group, and who should make that call?
- How large a stratified sample do you need before a difference between two groups is worth acting on?
- When adjusters review machine output, what would tell you they have stopped checking it carefully?
Learning Resources
Technical CASE STUDIES
China, India, United States, Indonesia, Pakistan, Nigeria, Brazil, Bangladesh, Russia, Japan, Mexico, Philippines, Vietnam, Egypt, Turkey, Germany, United Kingdom, France, Italy, Canada, South Korea, Australia, Spain.