AI Solution Development Case Studies
Technical5 Case StudiesUpdated Sep 2026
The cases below come from work that sits where product planning, data engineering, and daily operations meet. Each describes a team that had to build or repair an intelligent system under pressure, with limited budgets, uneven data, and colleagues who disagreed about what success meant. Organizations and people are invented, but the decision points are typical. Read for the reasoning rather than the outcome. When a choice looks obvious afterward, ask what the team knew at the time and who would absorb the cost of being wrong.
Case Study 1
Choosing one AI feature with a small team
Tessaly Field Services, a regional maintenance contractor with 420 technicians serving commercial HVAC and elevator sites.
The situation
Tessaly had three AI features on its list for the year. The first was a parts forecasting model that would predict which components each depot needed in the following two weeks, based on service history and seasonal failure patterns. The second was a draft generator for repair quotes, meant to cut the time office staff spent writing estimates after each visit. The third was a route optimizer that would rebuild technician schedules each morning. The product group had six engineers and one data specialist, and the operations director wanted a decision before the next budget review in the third week of the quarter.
The challenge
The three ideas had different kinds of evidence behind them. Parts forecasting had eighteen months of purchase records, but those records were spread across two inventory systems that disagreed on unit codes. Quote drafting had a clear pain point, since estimators reported losing around six hours a week to rewriting, yet nobody had measured error rates. The route optimizer looked impressive in a demo but depended on live telematics that only covered about half the fleet. Every option would consume most of the same engineers.
Analysis
The first question was which objective the spending should serve. Tessaly's stated goal for the year was to reduce repeat visits, since each second trip ties up a technician and a truck. Parts forecasting speaks directly to that goal, because a missing component is one of the most common causes of a return visit. Quote drafting serves revenue timing, which matters but does not touch the repeat-visit target. The route optimizer could help, but its benefit depended on telematics coverage that the company did not yet have. Setting the objective first removed two options from the argument before any cost comparison began.
Evidence and capacity were the harder constraints. The forecasting data was rich but messy, and a reconciliation step between the two inventory systems would take most of a month before any model could train on it. The team estimated that a pilot at two depots could be built in about six weeks if the data work was done first. Stakeholder impact also shaped the choice. Depot managers had been burned by a previous scheduling tool that promised savings and delivered confusion, so they wanted a pilot they could check against their own counts. Office staff would welcome quote drafting but would not be affected by a wrong forecast. Tessaly chose the option where failure was visible, bounded, and easy to reverse.
Actions taken
- Reconcile the two inventory systems on a shared unit code before any modeling starts, and publish the mismatch rate so depot managers can see progress.
- Run a six-week pilot at the two depots with the highest repeat visit rates, comparing forecast shortfalls against actual stockouts each week.
- Set a go or no go gate at the end of the pilot: forecast accuracy must beat the current reorder rule by a margin agreed with depot managers in advance.
- Park quote drafting in the backlog with a written owner and a date to revisit, so the office team knows the idea is deferred rather than dropped.
Results
After the reconciliation work, the mismatch rate between the two systems fell from roughly one in five part codes to under one in fifty, which was the most valuable outcome of the quarter even though no model shipped yet. The six-week pilot at two depots showed forecasts that beat the existing reorder rule on stockouts in most weeks, though not all. Leadership approved a wider rollout to the remaining depots, with a second review scheduled for the next budget cycle. Quote drafting went onto the roadmap for the following quarter, with a named owner in the office team.
Questions for discussion
- Which measure would have told the team early that the forecast data was not ready, and who should have owned it?
- How would the decision change if the depot managers had insisted on quote drafting first because its benefit was easier to see?
- What would a fair pilot comparison look like if the current reorder rule were itself poorly documented?
Case Study 2
Tracing repeated misroutes in a ticket classifier
Brightwater Systems Support, a managed IT provider handling about 9,000 tickets a month for mid-sized clients.
The situation
Brightwater's triage model read incoming support tickets and routed them to one of six resolver queues. When it went live, the team expected faster first responses, and the first month did show that. By the fourth month, however, the share of tickets that had to be rerouted by hand was climbing, and a visible group of clients were reporting that urgent outages sat in general queues for hours. The service delivery lead tracked breach counts every Monday, and breaches tied to misrouted tickets had become the most common reason for credits issued to clients.
The challenge
The recurring problem was not a single bad release. Misroutes clustered around certain ticket types, particularly those written in short messages from mobile devices, and around a newer client whose naming conventions for systems differed from the training data. Agents were correcting routes manually but logged the corrections inconsistently, so nobody could say whether the model or the people were the weak link. Any fix had to avoid adding review steps, because the queues were already staffed close to the limit.
Analysis
The business objective was clear enough: keep first response inside the contracted window and stop the credits. The evidence was less tidy. Brightwater logged routing outcomes in three places, and the reroute field in the ticket system was filled in only when an agent remembered to change it. The team therefore built a process map from intake to resolution and marked each handoff where a label could change. That map showed that the model's decision was often overwritten at a dispatch step, where a senior coordinator reassigned tickets by hand, and that those overrides were not fed back into training.
Testing causes meant separating three explanations and checking each against data. Short mobile messages were tested by sampling two hundred tickets from that channel and comparing the model's confidence with the outcome. Naming mismatches were tested by comparing the new client's system names with the training vocabulary, which showed that about a third of them never appeared in training. The third cause, overwritten labels, surprised people most, and it was the cheapest to measure. Stakeholder impact was uneven. Clients saw credits, coordinators saw extra work, and the model team saw accuracy figures that looked acceptable on paper. Each group needed a different measure before it would trust the result.
- Ticket intake Mobile and short messages arrive with thin context and naming gaps.
- Model routing Confidence is scored, but low confidence routes still reach live queues.
- Manual override Coordinators reassign tickets without a reason code, hiding the cause.
- Retraining batch Overrides return to training only when someone exports them by hand.
Actions taken
- Make every manual reroute write a structured reason code at the moment of change, so corrections can be counted and traced to a cause.
- Add the new client's system names to the training vocabulary and measure the misroute rate for that client weekly against a target of under five percent.
- Feed coordinator overrides back into a retraining batch every two weeks, and hold out one month of overrides to test whether accuracy improves on live traffic.
- Route ticket messages under a set length from mobile channels to a confidence check, and send low confidence items to a holding queue with a four-hour review limit.
Results
Reason codes went live in the second week of the new cycle, and within a month the team could say that roughly sixty percent of reroutes came from overwritten labels rather than model error. The vocabulary update cut misroutes for the new client from about one in four tickets to a little under one in ten. The confidence check added a small queue that cleared within its review limit most days. Breach counts dropped, and credits for misrouting fell in the following quarter. Some misroutes remained on mobile tickets, which the team accepted as a known residue while the holding queue was tested.
Questions for discussion
- Which of the three causes would you have tested first, and what evidence would have changed that order?
- How should the team weigh coordinator overrides as a training signal when the overrides themselves may reflect workload pressure?
- What would make a weekly misroute target misleading for a client with very few tickets?
Case Study 3
Settling a release dispute between two teams
Ridgeline Power Systems Software, a grid software unit serving 60 municipal utilities.
The situation
Ridgeline's field operations group wanted an anomaly model for transformer telemetry live before the summer peak, arguing that two outages in the previous season had been visible in sensor data hours before crews were called. The platform team, which owned the data pipeline, wanted a further month of testing and a documented rollback path. The compliance lead had a separate concern about whether alerts sent to utility operators would count as operational directives under customer contracts. Each group had a manager with a stake in the outcome, and the release calendar had no owner for the disagreement.
The challenge
The disagreement looked like a timing argument, but the underlying interests were different. Field operations cared about avoiding outages and was willing to accept false alarms if crews could dismiss them quickly. The platform team cared about not paging operators for noise, since a previous alerting change had flooded one control room. Compliance cared about contract language that nobody had reread in years. Each side presented its position as a safety or reliability requirement, which made it hard to see where they could move without losing face.
Analysis
The shared business objective was fewer unplanned outages during peak load, and that gave the negotiation a place to start. Ridgeline's evidence was a replay of the last season's telemetry, run against the proposed model. The replay showed the model would have flagged both earlier outages, but it also produced alerts at a rate that would have crossed operator fatigue thresholds. The replay did not settle the question on its own, because it used historical labels that were incomplete. What it did provide was a concrete artifact that each group could attack or defend, which moves a negotiation further than general talk about risk.
Teams grounded in AI Solution Development usually start by separating who decides from who must be consulted, and this case needed exactly that. Ridgeline's release policy gave the platform team sign-off on production data flows, but gave no one authority over the alert thresholds that operators would experience. Compliance could block a release but could not set thresholds. The practical agreement therefore split the decision: platform would own the rollback criteria, field operations would own threshold tuning within an agreed band, and compliance would review contract wording before the first live alert. Operators, who would bear the consequences, had no seat in the room, so the team invited two of them to the threshold review.
Actions taken
- Write down decision rights for each part of the release in one table, with a named owner for thresholds, rollback, and contract language.
- Agree on a threshold band with field operations and tune within it for four weeks using replayed alerts, with a fixed weekly review of false alarms.
- Ask compliance to mark which alert types, if any, count as directives, and route only non-directive alerts to operators during the first month.
- Hold a joint review at the end of the first month where platform, field operations, and two operators decide whether to widen, hold, or roll back.
Results
The release went ahead about three weeks later than field operations had wanted but earlier than the platform team had proposed. The threshold band held through the first month, and the weekly false alarm review cut nuisance alerts by roughly half after the second adjustment. Compliance confirmed that the first alert class was informational, which allowed the wider rollout to proceed on schedule. The two operators who joined the review became the most useful critics of later changes. Relations between the groups stayed strained for a while, but the decision table gave everyone a place to take a disagreement without restarting the argument.
Questions for discussion
- Which party in this dispute had the least formal authority, and how would you give that party a real role without slowing the release?
- If the replay had shown no difference between the old and new thresholds, which interests would have remained unresolved?
- How would you test whether the decision table is being used, rather than just filed?
Case Study 4
Activity rose while the targets kept slipping
Quillon Health Data, a company that builds document extraction tools for about 140 regional pharmacy chains.
The situation
Quillon's extraction model read scanned prescription claims and filled structured fields that billing staff then checked. Over the past two quarters, the team had shipped four model versions, processed a steadily rising volume of claims, and reported higher throughput at every monthly review. Yet the client-facing target, which capped fields needing correction at four percent, had been missed in five of the last six weeks. The program manager had a dashboard showing volume, model version, and average confidence, and nothing on it explained why corrections were rising.
The challenge
The metrics were tracking the wrong thing well. Volume was counted at intake, so a batch that failed silently still counted as processed. Average confidence was computed across all fields, which hid low confidence on dates and dosage, the fields most likely to need correction. Correction rates came from a sample of 50 claims per week, pulled by a person, and were reported after the following Tuesday. By then, the week's work had already moved on. Nobody had a defined threshold for when a number should trigger a change.
Analysis
The business objective was not throughput. Clients paid for claims that were right the first time, and rework cost them staff hours and delayed reimbursements. Once that was clear, the team could judge its metrics against the objective. Throughput was a legitimate operational measure, but it had been treated as a proxy for quality, and the evidence did not support that. A sample of 50 weekly claims is small, but the sample was drawn across the whole week, so it did reveal a trend. What it could not reveal was which fields drove the trend, because the review form recorded only a yes or no for each claim.
Constraints shaped what could be measured quickly. Field-level correction logs did exist in the review tool, but they were stored in a format the dashboard could not read without a manual export, and the reviewing staff were already at capacity. Stakeholder impact ran in two directions. Billing staff felt blamed for corrections they had not caused, which made them reluctant to report errors in detail. The model team felt that rising volume was evidence of progress and resisted a metric that made them look slower. The redesign therefore needed to make correction causes visible without turning the review into a verdict on people.
Actions taken
- Count a claim as processed only after its fields pass a validation check, so silent failures appear in the throughput number rather than hiding behind it.
- Report confidence and correction rates separately for each high risk field, starting with dates and dosage, and drop the blended average from the dashboard.
- Move the sample to a random draw of 200 claims per week, scored with a reason code, and publish results on Monday so the team can act within the same week.
- Set a written trigger that opens a review when any high risk field exceeds six percent correction for two consecutive weeks, with a named owner for the response.
Results
The validation step lowered reported throughput for the first two weeks, which was unwelcome but honest. Within the following month, the field-level view showed that dates caused most corrections, traced to a scanning setting that cut off the right edge of some forms. Fixing the scan setting brought the date correction rate down by more than half. The overall target was met in the eighth week after the change, and the trigger fired only once, which was the first time the team had seen a problem flagged before a client did. Billing staff began filing detailed reason codes without being asked.
Questions for discussion
- Which single metric would you remove from the current dashboard first, and what would you lose by doing so?
- How would you keep the reason code review from becoming a way to assign blame to billing staff?
- What threshold would you set for the trigger, and how would you check whether it is too sensitive or too slow?
Case Study 5
Shipping a faster model with hidden exposure
Oakmere Credit Systems, a software vendor that supplies loan scoring engines to 30 regional lenders.
The situation
A regional lending client asked Oakmere for a scoring update that would add mobile payment history as an input. The client's sales team had modeled the change on a backlog of applicants who were being declined for thin credit files, and early numbers suggested approvals in that group would rise noticeably within one quarter. The client's board wanted the change in place before the next lending season. Oakmere's engineering team had the model ready for review, and the sales director had already promised a launch date to two referral partners.
The challenge
The benefit was real, and so was the exposure. Payment history from mobile wallets is often thin, unevenly collected, and tied to demographics in ways that can correlate with protected characteristics. Oakmere had not yet tested whether the new input improved repayment prediction or simply reshuffled who was approved. The consent terms under which the data would be collected had been drafted for a different product. And the audit trail for scoring decisions could not yet show which inputs drove a particular denial.
Analysis
Oakmere's business objective was to keep its lender clients, and a flat or shrinking approval rate for thin-file applicants was a real threat to that. The short-term benefit was also honest: the client's pilot showed a meaningful lift in approvals, with default rates that looked stable over a short window. The problem was the length of the window. Default outcomes on loans of this type take months to show, and the pilot had covered only the first cohort. The evidence supported a limited test. It did not support a claim that the new input was fair or predictive across the whole applicant population, and the sales deck had said both.
Three options were on the table. Shipping as requested would lift approvals within weeks but would leave Oakmere answering for decisions it could not yet explain, and the client's regulator could ask for those explanations. Shipping with the new input held to a shadow mode for two cohorts would delay the lift but generate the fairness and accuracy evidence the claim needed. Declining the feature entirely would protect the company's position but could cost the client and push it to build the same input in-house with weaker controls. The third option kept more doors open, and the team recommended it, with its cost named plainly: thin-file applicants would wait longer for a decision.
- 1Pilot resultsApproval lift looked strong, but covered only the first cohort.
- 2Shadow modeInput is scored and logged without changing any decision.
- 3Fairness reviewResults are checked before any applicant sees the new input.
- 4Consent updateCollection stops for applicants who have not signed new terms.
- 5Limited releaseNarrower input ships with attribution on every denial.
Actions taken
- Run the mobile payment input in shadow mode for two full loan cohorts, logging what it would have changed without affecting any approval decision.
- Commission a fairness review of approval and denial rates across protected groups where lawful data exists, and publish the method before any results.
- Rewrite the consent language for the new data source with the client's counsel, and stop collecting the feed for any applicant who has not signed the updated terms.
- Add per-decision input attribution to the audit trail before go-live, so any denial can be explained to an applicant or regulator within a set period.
Results
The client accepted the shadow mode, though not without friction. The sales director had to walk back a launch date. Two cohorts later, the shadow data showed the input improved prediction for a subset of applicants but added little for the rest, and the fairness review flagged a gap that the team could partly close by changing how missing payment history was handled. The revised model went live in the following quarter with the narrower input, an updated consent form, and attribution on every denial. The approval lift was smaller than the original pitch, but the client kept the account.
Questions for discussion
- What would have to be true for the shadow period to be shortened without weakening the case for it?
- Who inside Oakmere should have authority to stop a client launch on fairness grounds, and what would make that person's position credible?
- How would you explain the smaller approval lift to a client who has already described the larger one to its board?
Learning Resources
Technical CASE STUDIES
China, India, United States, Indonesia, Pakistan, Nigeria, Brazil, Bangladesh, Russia, Japan, Mexico, Philippines, Vietnam, Egypt, Turkey, Germany, United Kingdom, France, Italy, Canada, South Korea, Australia, Spain.