IT Service Management Case Studies
Technical5 Case StudiesUpdated Sep 2026
These five cases come from operations teams that run internal platforms, where a slow checkout page or a delayed payroll file can reach a manager's desk quickly. Each one starts with a real pressure point, such as a failed release, a report that disagrees with the finance ledger, or a vendor tool that promised to cut ticket volume. The write-ups follow the same path: what the team saw, what limited their choices, who felt the effect, and what they changed afterward. The organizations are fictional, but the tradeoffs will feel familiar to anyone who has been on call.
Case Study 1
Checkout Latency Spike After a Friday Release
Larkspur Freight Digital, the internal platform group for a regional freight carrier, running order tracking and dispatch services
The situation
Larkspur Freight Digital runs the order tracking and dispatch services for a regional carrier with roughly 2,300 daily shipments. In the second week of the quarter, a release moved the tracking lookup to a new caching layer. Within the first hour after deployment, the dispatch board began showing stale delivery windows, and then response times for the tracking endpoint climbed past what the support desk could tolerate. Calls from dispatchers started arriving before the on-call engineer had acknowledged the page. The release had passed load testing at a fraction of production volume, and the rollback path had not been exercised since the caching layer was introduced.
The challenge
The team had to restore tracking for dispatchers without losing the release fixes that the carrier's billing reconciliation depended on. Rolling back meant reverting a schema change that had already written new cache keys, and nobody was certain whether the old code could read them. Mariam, the on-call lead, had to choose within the hour between rolling back, disabling cache reads, or scaling the origin service, while the operations director asked for a status update every fifteen minutes.
Analysis
Teams grounded in IT Service Management usually begin by writing down what the business is protecting before they touch the system. For Larkspur, the objective was narrow: dispatchers needed accurate delivery windows before the afternoon route plans were locked, and billing needed the release fixes intact. The evidence was uneven. Application logs showed cache timeouts clustering after the deployment, but the origin database graphs looked normal, which suggested the problem sat in the new layer rather than in capacity. The team treated that split as a hypothesis to test, not a conclusion. Speed mattered, but a wrong guess about capacity would have sent engineers scaling the wrong tier.
Practical limits set the order of work. Rollback was the cleanest option, but the schema change made it risky until the team confirmed the old reader ignored unknown keys. Disabling cache reads was faster, yet it would push roughly 40 percent more queries onto the origin database during peak dispatch, which the database team had already flagged as near its limit. Dispatchers lost trust in the board, the support desk absorbed repeated calls, and finance worried about reconciliation gaps. Mariam disabled cache reads for the affected endpoint, then checked the old code path against a copy of recent keys before approving the full rollback.
Actions taken
- Mariam set a fifteen-minute update cadence in the incident channel and assigned one engineer to keep the timeline current, so the operations director stopped asking for status in direct messages.
- The team disabled cache reads for the tracking endpoint, which restored accurate delivery windows within twenty minutes while the origin tier was scaled to absorb the extra queries.
- Before the full rollback, an engineer replayed a sample of recent cache keys through the previous reader in a staging copy, confirming it skipped unknown fields without failing.
- The team added a pre-release check that replays production-shaped traffic at full volume, and the rollback path is now exercised in every release rehearsal before a change is approved.
Results
Tracking latency returned to its normal range within the first afternoon, and the full rollback was never needed. Dispatch board staleness lasted about forty minutes in total, against an earlier estimate of two hours. The support desk logged 112 calls during the incident, most of them before the cache change took effect. The following release passed rehearsal with the rollback path exercised, and the team now tracks cache key versions as part of every schema review.
Questions for discussion
- Which evidence from the incident would you have wanted before choosing between rollback and a cache bypass, and how would you have gathered it within the hour?
- How should a team decide who owns status communication when the on-call lead is also the one making the technical call?
- What would a rehearsal need to include before you trust a rollback path that has not been exercised in production?
Case Study 2
Conflicting Weekly Reports From Mismatched Order Feeds
Tessaly Foods, a regional grocery distributor with three warehouses and a finance team that reports weekly to its board
The situation
Tessaly Foods distributes packaged goods to about 900 stores from three warehouses. Each Monday, the planning team and the finance team each presented a weekly fill rate, and for two consecutive weeks the numbers differed by almost four points. The planning figure came from the warehouse management system, while finance pulled shipments from the billing extract. Both teams trusted their own source and assumed the other had made an error. Nobody had a shared definition of a shipped unit, and the billing extract was refreshed on a different schedule. The disagreement surfaced in a board pack, which made it a governance problem before anyone had confirmed the cause.
The challenge
The analytics lead, Oluwaseun, needed to explain which number the board should use without pretending the other was wrong. The data platform team had no lineage diagram for the fill rate metric, and two earlier fixes had patched symptoms in the transformation layer without recording what they changed. The warehouse team was mid-cycle on a system upgrade and could not spare anyone for a full audit before the next board meeting.
Analysis
Practitioners who handle reporting disputes tend to separate the decision the business needs from the metric that happens to be on the slide. At Tessaly, the board needed a single fill rate to decide whether to add weekend trucks, and the planning team needed a figure that matched what drivers actually loaded. The evidence was partial. Row counts matched between the warehouse system and the billing extract for most days, but the difference clustered around partial shipments and returns that were posted on different dates. That pattern pointed toward a definition problem rather than a broken pipeline, which changed where the team looked first.
Constraints shaped the response more than the data did. The board pack was already circulated, so any correction had to be explained in a way that did not undermine trust in either team. The warehouse upgrade meant the source system could not be queried for historical snapshots until the cutover finished, so the team could only validate against exports taken in the previous fortnight. Planners worried that a changed definition would make their weekly targets look worse overnight, and finance worried that restating the prior quarter would trigger a review of the forecast. Both concerns were legitimate, and the team treated them as design requirements rather than objections.
- Define Agree the event, timestamp, and treatment of partial shipments.
- Trace Map each join and transformation back to its source extract.
- Reconcile Compare counts and totals before every weekly publish.
- Monitor Alert owners when definitions or source volumes shift.
Actions taken
- Oluwaseun published a one-page definition of shipped fill rate, naming the event that counts, the date it is stamped, and how partial shipments are split, and the board agreed to adopt it.
- The data engineers traced the metric from the dashboard back to both source extracts, recording each transformation step and the owner of every join in a lineage map kept beside the pipeline code.
- Each transformation now carries reconciliation checks that compare row counts and value totals against the warehouse snapshot, and any mismatch above a set tolerance blocks the weekly publish.
- The team restated the previous eight weeks in a parallel table, labeled clearly, and gave planners and finance a shared review window before the figures replaced the originals in the board pack.
Results
Within the cycle after the definition was agreed, the two weekly figures matched to within a fraction of a point, and the remaining gap was explained by partial shipments that the definition now handled. The board accepted the restated eight weeks with one question about a single depot, which the warehouse team resolved from its own logs. The reconciliation checks blocked one publish in the following month when a late billing extract arrived with duplicate lines, and the data team fixed it before planners saw the error. Monitoring now flags definition changes to the owners of every downstream report.
Questions for discussion
- Where in your own reporting chain would a shared metric definition be hardest to agree on, and who would you need in the room?
- When a correction threatens a figure already in front of the board, what sequence of communication protects both the analysts and the decision?
- Which reconciliation tolerance would you set first, and how would you revisit it once the source system stops changing?
Case Study 3
Suspicious Service Account Activity Before Payroll
Ridgeline Community Credit, a credit union serving about 48,000 members with a small internal security group
The situation
Ridgeline Community Credit serves about 48,000 members and runs its own identity platform for staff access to loan systems. In the middle of a Tuesday, the security analyst on watch, Dev, noticed a service account authenticating from an address in a region where no staff worked, followed by a burst of directory queries. The account belonged to a scheduled reporting job that had been rotated two weeks earlier. Within the same hour, two employees reported password reset emails they did not request. Nothing had been changed in the loan system yet, but the payroll data export was scheduled to run that evening.
The challenge
Dev had authority to isolate the service account but not to disable staff sessions, and the decision on whether to suspend the payroll export belonged to the chief operating officer, Priya, who was on a flight until the early evening. The team had no confirmed scope. The logs from the identity provider covered thirty days, but the directory query logs were only kept for seven, and two of the affected staff had used the same mobile device for work and personal email.
Analysis
Responders on small security teams usually separate what they can prove from what they suspect. Ridgeline's objectives were concrete: protect member records, keep the payroll export from running against a compromised directory, and avoid a disruptive lockout of every staff member during a busy close week. The evidence pointed in more than one direction. The rotated service account was a strong signal, but the password reset emails could come from a phishing campaign aimed at staff, a separate compromise, or a misfiring alert. Dev wrote down each observation with its source and timestamp, and marked which ones a second log confirmed.
Containment options each carried a cost that someone else would pay. Disabling the service account was clean, but the reporting job fed a regulator-facing dashboard, so its failure would surface to examiners within days. Forcing a reset for all staff would stop the attacker only if credentials were the sole path, and it would stall loan approvals during the close. Priya's approval was needed for the payroll decision, and she could give it only once she landed. In the meantime, the support desk fielded calls from members who saw unusual sign-in notices, and the team kept the message to members factual and brief until scope was known.
Actions taken
- Dev isolated the service account and revoked its active tokens, then captured the directory and identity logs to a write-once store before anyone changed a password.
- Marcus, the incident lead, approved a narrow scope: force resets only for the two employees who reported emails, and hold the payroll export until the chief operating officer confirmed by phone.
- The team reviewed every sign-in from the affected account over thirty days against a baseline of known devices and locations, and recorded which events were confirmed malicious, which were unexplained, and which were benign.
- Each step taken was logged with the time, the person authorizing it, and the reason, and the written record was reviewed by the compliance officer before the incident was closed.
Results
The review found the service account had been used from one unfamiliar address on four occasions, all tied to a stolen session token from a phishing page, and no evidence that member records were read. Two staff accounts were reset, and the payroll export ran the next morning after the chief operating officer approved it from her hotel. The team now rotates service credentials on a schedule tied to the reporting cycle and requires token binding for accounts with directory read rights.
Questions for discussion
- What would have changed your containment choice if the payroll export had been scheduled for the following week instead?
- How should a small security group decide when to act on a signal it has not yet confirmed, and who should be told at each step?
- Which of the logs in this case would you have wanted to keep longer, and what retention decision would you defend to an examiner?
Case Study 4
Choosing a Read Path for Inventory Lookups
Corvane Retail Systems, a point-of-sale vendor whose cloud service handles inventory lookups for about 1,400 independent stores
The situation
Corvane Retail Systems sells a point-of-sale platform to independent stores, and its inventory lookup service now handles about 1,400 stores during weekday peaks. Store clerks expect a stock check in under a second, and the current design queries a central relational database for every request. Over the past two quarters, request volume roughly doubled as stores added online order pickup, and the database began to show lock waits during the lunch rush. Engineering had three candidate designs on the table: scale the existing database vertically, add a regional read cache with replication, or move lookups to an event-fed store that each region keeps locally.
The challenge
The platform lead, Teodor, had to recommend one design to a steering group that cared about three things: keeping stock counts accurate enough that clerks did not sell items that were already gone, holding infrastructure spend roughly flat for the year, and avoiding a migration that would disrupt stores during the holiday trading period. Each option promised a different balance. Nobody on the team could say with confidence how much staleness the store managers would accept.
Analysis
Architects on growing platforms tend to start with the tolerable failure, not the preferred architecture. For Corvane, a stock count that lagged by a few seconds was acceptable, while a count that stayed wrong for an hour after a delivery was not. The load data showed most lookups were repeats for the same SKUs within a store, which made caching attractive, but write bursts during receiving were concentrated in short windows. Those figures came from one quarter of internal traffic, so the team avoided extending them to stores with unusual patterns. The evidence supported a read path that tolerates small delays but must reconcile quickly after receiving events.
Constraints narrowed the field faster than the performance numbers did. Scaling the existing database kept the security model unchanged and avoided new data stores, but it did little for the lock waits during receiving, and the vendor contract capped the largest instance size. A regional cache needed replication across regions, which meant customer payment-adjacent data would move, so the security review would take longer and require encryption controls the team had not yet built. The event-fed store was the most resilient option, but it needed a stream platform the operations group had never run, and managers would see changes only after the regional copy caught up.
Actions taken
- Teodor ran a two-week proof on a copy of the inventory data, replaying a recorded receiving window to measure how far stale counts drifted under each design.
- The team chose the regional cache with a reconciliation job that replays receiving events within seconds, and kept the central database as the system of record, leaving audit and security controls unchanged.
- Store managers received a clear rule: a count is trusted only after the receiving event has reconciled, and the till displays a small marker until that happens.
- The security group reviewed the cache design before build, adding field-level encryption for the few attributes that could identify a customer, and set a release gate tied to the lock wait metric.
Results
Stock count drift during receiving stayed under the agreed threshold in every replay of the busiest window, and lock waits on the central database dropped sharply once lookups moved to the regional cache. Holiday trading ran through the migration without a store-facing incident, though two regions needed a second reconciliation pass after a supplier sent duplicate receiving files. Infrastructure spend rose modestly, below the increase the vertical option would have required. The event-fed store remains on the roadmap for a later stage, once the stream platform has been run in a smaller setting.
Questions for discussion
- How would you have tested the staleness tolerance with store managers before committing to a design that depends on it?
- When a proof of concept uses recorded traffic, which kinds of load would it still miss, and how would you find them?
- At what point would you accept a design that is harder to operate because it removes a bottleneck the team already understands?
Case Study 5
Piloting an Automated Claim Triage Assistant
Halvorsen Insurance Services, a claims administrator with about 260 staff handling property and auto policies for regional carriers
The situation
Halvorsen Insurance Services processes property and auto claims for regional carriers, and its intake queue held roughly 6,500 open files at the start of the quarter. A vendor offered an automated triage assistant that reads incoming claim notes, assigns a priority and a routing category, and drafts a first acknowledgment letter. The vendor's pitch centered on halving the time claims spent in intake, and the operations director, Rhonda, was under pressure to show progress before the renewal discussions. The vendor's demonstration set and accuracy summary came from its own test data, not Halvorsen's files, which include scanned adjuster notes and a long tail of unusual claim types.
The challenge
Rhonda needed a recommendation that would survive scrutiny from the carriers, who audit how claims are handled and can object if an automated step changes what a customer receives. The claims leads were worried about misrouted severe losses, and two of them had seen earlier tools fail quietly on edge cases. The team had to define what accuracy meant for a triage tool, decide who checked its output, and set a rollout that could be stopped within a day if something went wrong.
Analysis
Teams evaluating automation usually define the failure they cannot tolerate before they score the tool. At Halvorsen, the objectives were efficiency and consistency, but a misrouted severe loss costs far more than a slow acknowledgment, so the team ranked error types by consequence. The vendor's figures were a single accuracy number on clean text, which hid the outcomes that mattered. The team asked for a sample of recent claims labeled by two adjusters, drawn across claim types and carriers, and planned to compare the tool's routing against those labels, paying particular attention to severe losses and to claims filed by older policyholders.
Regulation mattered as much as the technology. Carrier contracts require a documented human decision on any denial or reduction, so the assistant could suggest a route but could not close a file. Adjusters would absorb any extra review work, and they had already said that a tool which generated more rework than it saved would be ignored within weeks. Policyholders would not see the tool, but they would feel a misrouted claim as a delay in first contact, and some carriers track that delay as a service measure. The vendor's data handling terms required a review before any claim text left Halvorsen's environment, adding weeks to the timeline.
- 1Blind labelingAdjusters label a stratified sample before seeing output.
- 2Threshold checkSevere-loss misroutes must stay under the agreed limit.
- 3Limited pilotOne carrier, drafting and suggested routes only.
- 4Daily samplingClaims lead reviews a random set and logs each miss.
- 5Pause triggerSame-day halt if severe misses or rework exceed limits.
Actions taken
- Rhonda approved a blind test in which two adjusters labeled a stratified sample of recent claims before seeing the tool's output, and the team measured agreement by claim type and carrier.
- The team set success as fewer than one severe loss misrouted per thousand reviewed claims, plus an agreed floor for every claim type, and recorded both before the test began.
- The assistant was limited to suggesting a route and drafting an acknowledgment that an adjuster must approve, with a daily sample reviewed by a claims lead and a one-click switch back to manual routing.
- The rollout started with one carrier's auto claims for a fixed period, and the team agreed that a severe-loss miss or rework above an agreed level would pause it the same day.
Results
The blind test showed strong agreement on routine claims, but the assistant under-routed a small set of severe losses filed with incomplete notes. Those misses were enough to change the decision. Halvorsen kept the assistant for drafting acknowledgments only, where errors were easy to catch, and routing stayed with adjusters until the severe-loss gap was closed. Intake time for acknowledged claims fell noticeably in the pilot, and the claims leads reported fewer repeat calls from carriers. The rollout to a second carrier is scheduled for the next review, conditional on the same thresholds.
Questions for discussion
- Which claim attributes would you have added to the blind sample to surface bias before the pilot began?
- How should a team decide whether a drafting feature that errs rarely still needs the same oversight as a routing feature?
- What would you need to see in rework data before you would call an automation tool worth its maintenance cost?
Learning Resources
Technical CASE STUDIES
China, India, United States, Indonesia, Pakistan, Nigeria, Brazil, Bangladesh, Russia, Japan, Mexico, Philippines, Vietnam, Egypt, Turkey, Germany, United Kingdom, France, Italy, Canada, South Korea, Australia, Spain.