Site Reliability Engineering Case Studies
Technical5 Case StudiesUpdated Sep 2026
Reliability work sits where software design meets the daily behavior of real users, and the five cases here reflect that. Each starts from a situation an operations group could plausibly face: a release that degrades service, reports that disagree, a suspected intrusion, a growth problem that forces a design choice, and an automation proposal with unclear failure modes. Every analysis looks at the same four things: design constraints, operating data, quality tolerances, and maintenance records. All organizations are fictional, and all figures belong to them alone.
Case Study 1
Release rewrites holds during a stadium on-sale
Larkspur Ticketing, a regional seller of event tickets whose checkout service handles seat holds for 340 venues.
The situation
Larkspur's checkout service had run without trouble for most of a year when the on-sale for a large stadium tour opened. The release that morning changed how seat holds were written to the cache, and it passed staging because staging held only a few hundred test seats. Within twenty minutes, p99 latency on the hold endpoint climbed from 180 milliseconds to over 4,000, and the monthly error budget was spent before the on-call lead finished the first dashboard. Customers saw spinning wheels and duplicate holds, and support volume tripled in the same window.
The challenge
Rollback was the obvious move, but the release had also run a one-time migration that rewrote hold records, so reverting code alone could leave half the holds in a format the old version could not read. Ines, the on-call lead, needed to decide whether to roll back, pause sales, or patch forward, while the incident channel filled with venue managers asking for an estimate. She had about fifteen minutes before the queue of waiting buyers would exceed what the payment provider could absorb.
Analysis
The first useful question was what the design allowed to be reversed. Teams grounded in Site Reliability Engineering tend to map constraints before touching a rollback button, and here the constraint was plain: the migration had written hold records that the previous build could not parse. Error rates were flat for requests that never touched migrated rows, while failures clustered on holds created after the cutover. The tolerance allowed 0.1 percent of holds to end in an unknown state, and the measured figure passed that within the first half hour.
The maintenance records told a second story. Two earlier releases had carried schema changes that were flagged in review but never given a staged rollout, and the runbook for migrations was last revised long before the current team joined. Once the incident was contained, the team reviewed every change that had touched the hold path over the previous year and found three with no canary step at all. Prevention meant three concrete changes: staging seeded with production-scale seat counts, a rule that no migration ships without a documented reverse path, and a canary gate that compares hold error rates against the tolerance before traffic widens.
- Minute zeroMigration rewrites hold records in place.
- Minute twentyp99 passes 4,000 milliseconds; error budget spent.
- Minute thirtyFlag routes new writes to old format; sales paused.
- The next day2,300 records checked against payment records.
- Following quarterRollouts halt when hold errors exceed tolerance.
Actions taken
- Ines rolled the release back for new holds only, using a feature flag that routed writes to the old format, and paused the on-sale for eleven minutes while the queue drained.
- A small crew reconciled the 2,300 ambiguous holds against payment records, releasing unpaid seats and confirming paid ones before buyers were contacted.
- Support received a scripted status note explaining the delay, and venue managers got a ten-minute update cadence instead of ad hoc replies.
- The team added a canary gate that halts a rollout when hold error rates exceed the tolerance for five consecutive minutes, and wrote the reverse path into the migration template.
Results
Service recovered for all new holds within the hour, and the unknown-state rate returned under the tolerance before the tour's second on-sale window opened. The reconciliation closed 2,300 records in about a day, with 41 buyers needing manual seat changes. Support volume returned to its usual level by the end of the week. In the following quarter, the canary gate stopped two releases that would have repeated the pattern, and the staging environment now carries production-scale seat counts. The error budget for the month was not recovered, and the team reported that honestly to the business.
Questions for discussion
- Which constraints on the hold data should have been written down before the migration was approved, and who should have owned that list?
- How would the team's decision change if the reverse path had required the payment provider to reissue confirmations?
- What tolerance would you set for unknown-state records during an on-sale, and how often would you revisit it?
Case Study 2
Two dashboards disagree over the same stores
Brightwater Grocers, a regional grocery chain with 42 stores that sets weekly orders for its distribution centers from a shared forecast.
The situation
Brightwater's merchandising group had two dashboards that reported weekly sales for the same stores, and they disagreed by between 3 and 9 percent depending on the week. Regional managers were splitting the difference by hand before placing orders, which meant the forecast was no longer the thing driving replenishment. The discrepancy first surfaced when a produce buyer noticed that one distribution center had received stock for stores that were already closed for renovation. Nobody could say which dashboard was right, and the pipeline behind both had been built by three different teams over several years.
The challenge
The forecast model itself was stable, so the problem sat upstream. Store identifiers were stored in two formats, a point-of-sale feed arrived late for about one store in twelve, and a currency conversion step had been added for a pilot that was never extended. Anjali, the data platform lead, had to establish which numbers were trustworthy enough to act on before the next order cycle, which would run in the second week, and she had no shared definition of what counted as a complete week.
Analysis
Few of the constraints were exotic, but they shaped every later choice. Orders had to be placed on a fixed schedule, so each store needed a complete week of sales before the cutoff, and a late feed could not simply be dropped without a rule for filling the gap. Lineage tracing started at the order file and walked backward through each join. The most damaging fault was the store key, since a join on mismatched identifiers returned fewer rows without raising any error. Row counts matched at the source, but fell short after the join for stores using the older identifier format.
Quality tolerances had never been written down, which is why the two dashboards could both look reasonable. Anjali proposed three thresholds: a store-week counts as complete only when at least 98 percent of expected transactions arrive, reconciled totals may differ from source by no more than half a percent, and any store missing from the join is listed on the morning report. The maintenance records explained the drift. The currency step had been disabled in a patch but its code path still ran for a subset of stores, and nobody had reviewed the change log since it was written.
Actions taken
- Anjali mapped every store identifier to a single canonical key, keeping the legacy format in a lookup table so historical reports could still be reproduced.
- The team wrote row-count checks at each join, failing the build when output falls below the input count for any store with an active sales record.
- A reconciliation job compares each store-week against the source totals every morning and publishes any gap above the tolerance to the merchandising channel before orders are built.
- The disabled currency step was removed, and the change log was reviewed with the two teams that had shipped code into the pipeline, so ownership of each stage is written down.
Results
The orders for the second week were built from reconciled totals, and the regional managers stopped adjusting numbers by hand. The join fix recovered about 4,700 store-week rows that had been silently dropped over the prior months. Two stores that had been carrying phantom stock were corrected, and the produce buyer's original concern turned out to be a symptom of a wider gap. Reports from the two old dashboards were retired, and a single published table now carries the reconciliation status beside every figure. Some historical forecasts remain uncertain, and the team labeled them rather than guessing.
Questions for discussion
- Which of the three tolerances would your merchandising group resist most, and what evidence would change its mind?
- If the store key had been correct but the feed arrived late, how would the morning report need to treat an incomplete week?
- Who should own the canonical store mapping after the pipeline team changes, and how would you know it had drifted again?
Case Study 3
Service account pulls shipments at odd hours
Harrow Freight Logistics, a mid-sized carrier that runs its own dispatch and tracking systems across 60 depots.
The situation
An analyst in the security operations group noticed that a service account used by the tracking API had authenticated from an address block it had never used before, and then requested a large number of shipment records in the early hours. The account belonged to a reporting integration that ran on a fixed schedule, so the timing looked ordinary at first. The volume did not. Over about forty minutes, the account pulled more records than the previous ninety days combined, and the requests were spread across depots that the integration never touched. Logs from the identity provider showed one prior failed login from the same range.
The challenge
Containment could cost the business something real. Revoking the account would stop the pulls but also halt dispatch exports that the depots depended on every morning, and the depots were the customer-facing side of the company. Nasra, the incident commander, had to decide how much of the pipeline to freeze, whether to preserve the host images before rotating credentials, and who outside the security group needed to know. The evidence was also perishable, since the cloud logs rolled over on a short retention window.
Analysis
The integration's design shaped the response more than the threat did. The tracking API used long-lived service credentials, a choice made to keep the nightly reporting job simple, and the credentials could not be scoped to individual depots without a rewrite that had been deferred twice. Operating data gave the scope. Request logs were joined against normal baselines for the account, and the anomaly was not the rate alone but the combination of new source range, new depot set, and an unusually wide query window. The team treated every credential issued to that integration as possibly exposed until the scope was confirmed.
Tolerance in incident work is less about percentages than about what the team is willing to be wrong about. Nasra set a rule that no step could destroy evidence that the incident team had not yet copied, and that any action affecting customer-facing exports needed sign-off from the operations director. The maintenance records were useful for a different reason. The last rotation of the integration's credentials was recorded eleven months earlier, and the runbook said rotation should happen quarterly, so the gap itself became a finding. Every step was logged with its authorizer, and the log was kept outside the systems under review.
- Scope the exposure Map each token, depot, and read against legitimate use.
- Preserve evidence Copy images and logs with hashes before any rotation.
- Contain with approval Revoke sessions and gate exports under written operator sign-off.
- Record and remediate Log each authorized step and schedule per-depot scoping work.
Actions taken
- Nasra revoked the service account's active sessions and replaced its credentials, after the host images and log exports had been copied to the forensic store with hashes recorded.
- Dispatch exports were moved to a manual queue that a named operator released each morning, with the operations director approving each depot batch in writing.
- Scope was narrowed by querying the identity logs for every token issued to the integration, then matching each shipment read against the depots the job legitimately serves.
- The team opened a remediation item to scope credentials per depot, set a rotation alert against the runbook interval, and added the new source range to the blocked list pending review.
Results
The review found that the credentials had been used by one unauthorized party, who had read roughly 38,000 shipment records before access was cut. The prior failed login was traced to a credential-stuffing attempt that had not succeeded. Dispatch resumed for all depots within the same working day, with the manual queue running for only the first cycle. Customer notification was limited to the shipments confirmed as read, and the legal team reviewed the wording before it went out. The per-depot scoping work is still in progress, and the team has not claimed the problem is closed.
Questions for discussion
- Which evidence would you copy first when the logs are short on retention, and who should approve that copy?
- How would the containment plan change if revoking the account also stopped a process that could not be paused safely?
- What rotation interval would you commit to for long-lived credentials, and what would make you shorten it?
Case Study 4
Cells or regions for a growing booking service
Tallowmere Scheduling, a company that books appointments for 27 outpatient clinics and sells its booking API to partner practices.
The situation
The booking service had been a single regional deployment for several years, and it worked well until partner practices began sending bulk schedule imports during the morning clinic rush. Read latency for availability searches rose from about 90 milliseconds to over 1,200 during those windows, and the database team had already added two larger instances without lasting relief. Nobody had yet measured how much of the slowdown came from the imports rather than from patient searches. The product group wanted a second region for failover and a faster search experience for patients, and the finance group had a fixed ceiling on infrastructure spending for the year.
The challenge
The team had three realistic designs. A read replica set in the existing region was cheap and fast to build but did nothing for regional failure. An active-active deployment across two regions gave the best failover story but would roughly double the database licensing and force the team to resolve write conflicts in appointment slots, where a double booking is a clinical scheduling error. A cell-based split by clinic group sat between them. Sani, the architect, had to recommend one before the next budget review.
Analysis
The constraints were not negotiable on three points. Patient data had to stay in the region where the clinic was licensed, so no design could place a full copy of every record in an arbitrary location. Booking writes had to be strongly consistent within a clinic. Reads could tolerate a short lag, but a search that offered a slot already taken would generate a support call and sometimes a missed visit. Operating data from the previous busy period showed that 71 percent of search traffic came from a small set of partner practices, which made a cell split by clinic group more attractive than a uniform global spread.
Tolerances set the real ranking. The team agreed that search could be up to five seconds stale, that a booking confirmation had to be correct on the first attempt, and that no more than one double booking in a million writes was acceptable, a strict bar for a system with no safe way to reconcile after the fact. The maintenance records pointed to a cost the active-active option hid. Two previous cross-region experiments had been abandoned because the failover runbook was never rehearsed, and the on-call rotation had no one who had restored a database in another region. Operational readiness, not raw capacity, was the binding limit.
Actions taken
- Sani chose the cell-based design for the two busiest partner groups first, leaving smaller clinics on the existing single region until the cell routing was proven.
- Each cell keeps its own write path and a read replica in its licensed region, so a booking never crosses a boundary that the consent terms forbid.
- Search moves to a shared index with a stated staleness bound of five seconds, and the search service sets a visible notice when the index lags beyond that bound.
- Before any cell goes live, the team runs a full regional failover drill, restores a database in the standby region, and records the time to first successful booking.
Results
The cell design went into service for the two busiest groups in the fourth month of the project, and search latency during the morning window settled near 260 milliseconds. Monthly infrastructure spend rose by about 18 percent, which was inside the ceiling but above the first estimate. The first failover drill took 47 minutes to reach a first successful booking, against a target of fifteen, and the team spent the following weeks cutting that time down. Smaller clinics moved over more slowly than planned. The team set aside full active-active, and the decision was recorded with its tolerances.
Questions for discussion
- Which of the three tolerances would you revisit first if the partner practices asked for real-time search?
- How would the cell boundaries need to change if one large partner operated clinics in two licensed regions?
- What would the team need to see in drill results before it would trust a cross-region design again?
Case Study 5
Automated claims triage promises speed with risk
Wexford Claims Services, a third-party administrator that processes insurance claims for 14 employer plans.
The situation
Wexford's claims intake had grown faster than its review staff, and the average time from submission to first decision had stretched past the contractual limit. A vendor offered an automated triage model that would sort incoming claims into fast-track, standard, and manual review, and would auto-approve the fast-track group. The vendor's demonstration showed strong accuracy on a sample drawn from the last batch of clean records. Nobody at Wexford had checked how the model behaved on the messy claims that made up most of the backlog, or on claims filed in languages other than English.
The challenge
The question was not whether the model was accurate on average but where its errors fell. A wrong fast-track decision on a legitimate claim would delay payment and generate an appeal, while a wrong standard routing would cost only time. Lorcan, the operations manager, had to decide whether to pilot the tool, what it must prove before touching live claims, and who would hold the authority to stop it. A penalty clause for late decisions gave the vendor a reason to push for rapid adoption.
Analysis
The claims themselves set the limits. Each claim carries a benefit category, a policy version, and attachments that arrive in inconsistent formats, and the model had been trained on a clean export that stripped most of those features. Operating data from the backlog showed where the risk concentrated. Claims with attachments scanned at low resolution, and claims that changed policy version mid-cycle, made up a small share of volume but a large share of later reversals. Any evaluation had to run on a stratified sample that reflected the backlog, not the demonstration set, and report error rates by segment.
Tolerances had to be written as asymmetric limits. Wexford agreed that no fast-track approval could be wrong on a legitimate claim more than 0.5 percent of the time in any segment, that every auto-approved claim would be sampled for review at a rate of 1 in 20, and that the model would be restricted to the two lowest-risk benefit categories until those limits held for a full cycle. The maintenance records added a practical constraint: the vendor's last three model updates had changed routing decisions without a changelog entry, so any new version would be treated as a new system and re-tested from the start.
Actions taken
- Lorcan ran a shadow pilot for one full cycle, with the model scoring every claim while human reviewers made the decision, and compared routing against outcomes.
- Every segment was scored on false fast-track rate, appeal rate, and time to decision, with the tolerances written into the pilot report before any results were viewed.
- Auto-approval was enabled for the two lowest-risk categories only, with a named adjuster able to reverse any decision and a one-click switch that returned all claims to manual review.
- The vendor was asked to supply a changelog and a regression suite for each model version, and no update reached live claims until the suite ran against the segment sample.
Results
The pilot found that the model was reliable on clean claims but produced false fast-track decisions at 2.8 percent in segments with low-resolution attachments, well above the tolerance. Those segments were removed from auto-approval, and the model was retained for standard routing only. Average time to first decision fell by roughly a third in the restricted categories, and appeal volume did not rise. The vendor added a changelog after the second update, and the regression suite caught one routing regression before it reached live claims. The contract penalty was renegotiated to exclude decisions delayed by pending review.
Questions for discussion
- Which segment would you exclude first if the pilot had shown acceptable error rates overall but failures concentrated in one benefit category?
- How should the sampling rate for auto-approved claims change after the model has held its limits for several cycles?
- What would make you accept a vendor's refusal to share training data, and what would you require in its place?
Learning Resources
Technical CASE STUDIES
China, India, United States, Indonesia, Pakistan, Nigeria, Brazil, Bangladesh, Russia, Japan, Mexico, Philippines, Vietnam, Egypt, Turkey, Germany, United Kingdom, France, Italy, Canada, South Korea, Australia, Spain.