Infrastructure as Code Case Studies
Technical5 Case StudiesUpdated Sep 2026
Infrastructure as code treats servers, networks, permissions and monitoring rules as versioned text that can be reviewed, tested and rebuilt. The five cases below come from teams that adopted it under pressure, usually because a manual change caused an outage, a report disagreed with another report, or a vendor promised too much. Each case follows the same arc: the situation, the constraint that made it hard, two analysis paragraphs, the actions taken, and what changed afterward. Read them for the tradeoffs more than the tools, since the questions about evidence, ownership and blast radius hold steady.
Case Study 1
Checkout slowdown during a ticket onsale
Harbor Lane Ticketing, a regional online seller of event tickets that runs checkout across two cloud regions.
The situation
Harbor Lane sells tickets for regional concerts and sports events. Its checkout service, payment connector and cache tier are all declared in a shared repository of Terraform modules, and every change goes through a pull request and a plan review. In the third week of the quarter, a release tuned the connection pool for the payment connector and also changed the instance class of the session cache. The plan looked clean, and the reviewer approved it in a few minutes. Less than an hour after a high-demand onsale opened, checkout latency climbed from under one second to more than fifteen seconds, and roughly one payment attempt in five timed out.
The challenge
Priya, the platform lead, had to decide whether to roll back the whole release or only the connection pool setting. Rolling back everything would also undo a security patch on the cache hosts, which the compliance team had asked for. Marcus, the on-call engineer, could see the latency graphs but not which change caused them. The incident channel filled with guesses. Dana, the support lead, was taking calls from fans who had been charged twice or not at all, and she needed a timeline she could give them.
Analysis
Teams grounded in Infrastructure as Code usually start by asking what the business is protecting before they touch a single resource. Here the objective was plain: keep the onsale completing payments while the fault was isolated. Revenue was the obvious stake, but so was the chance that customers would pay twice and lose trust in the next sale. The evidence was better than it first looked. The plan output, the state file history and the deploy log showed exactly which attributes had moved, and that narrowed the search to two changes instead of the whole release.
The constraints shaped the response as much as the diagnosis did. The onsale could not be paused without refunding a large queue of buyers, so the team had to restore service under live traffic. Rolling back through the pipeline would take about twelve minutes, while a targeted edit to the pool size could be applied directly, but that edit would drift from the repository and had to be reconciled afterward. Stakeholders cared about different things: support wanted accurate messages, compliance wanted the patch kept, and finance wanted no duplicate charges reported in the morning.
- 1Detect latency spikeDashboards showed checkout p95 above fifteen seconds within minutes of the onsale.
- 2Narrow the change setPlan output and state history isolated two attributes out of the full release.
- 3Revert the pool settingA targeted edit restored normal latency without rolling back the security patch.
- 4Reconcile through reviewThe reverted value was committed, planned and approved so the repository matched production again.
- 5Add a plan gateChanges to connection limits now need a second approver with load test history.
Actions taken
- Marcus reverted the connection pool attribute with a targeted change, then confirmed latency returned under two seconds before anyone touched the cache tier.
- Priya froze the release pipeline and opened a reconciliation branch so the reverted value would be committed, reviewed and planned instead of living only in the console.
- Dana's team posted a short status note every fifteen minutes and held a separate script for refunds, so agents never promised outcomes the engineers had not confirmed.
- The team added a plan gate that flags changes to connection limits and instance classes, requiring a second approver who has run the load test suite for that service.
Results
Service was restored in about forty minutes, and checkout latency returned to its normal range before the onsale queue cleared. Harbor Lane refunded the duplicate charges identified in the morning, which came to a small number of orders and was handled without a formal complaint escalation. The reconciliation branch went through the normal review, and the next planned release showed no drift. The plan gate has since caught two proposed changes to connection limits that would have repeated the same pattern. Priya now treats any plan that touches concurrency settings as a release risk, not a formality.
Questions for discussion
- Which evidence would have told the team about the pool change before the onsale opened, and who should have been responsible for collecting it?
- When a rollback would undo a compliance requirement, how should the decision be recorded so the next review can see the tradeoff?
- How would you decide whether a targeted console edit is acceptable during an incident, and what must happen before it becomes permanent?
Case Study 2
Conflicting on-time figures across weekly reports
Lattice Freight Partners, a mid-sized freight forwarder that consolidates shipment events from five carrier feeds.
The situation
Lattice runs a warehouse of shipment events pulled from five carrier feeds, transforms them in scheduled jobs, and publishes a set of weekly dashboards that account managers use with customers. Each feed uses its own status codes, and the transformation layer maps them into a single delivered flag. In the second week of the quarter, the operations director noticed that the on-time rate for one regional lane was four points higher on the dashboard than in the carrier's own statement. Two account managers had already quoted the higher number to a client.
The challenge
Sanjay, the data engineering lead, could reproduce the gap but not explain it. The raw feed had both a delivered code and a delivered-with-exception code, and the mapping treated the second as delivered in one job and as pending in another. Renata, the finance analyst, needed a single number for a contract review in the next fortnight. Several teams had built their own adjustments in spreadsheets, and nobody could say which version was authoritative.
Analysis
The business objective here was not to produce the most accurate possible number. It was to produce one number that every team would defend in the same meeting. Account managers needed figures they could repeat to clients without retracting them. Finance needed traceability for contract terms. Evidence was uneven: the raw carrier files were retained, the job logs showed each transformation run, and the dashboard queries were versioned, but the spreadsheet adjustments lived in personal folders. The first task was to rebuild the lineage from raw records to the published figure.
Constraints ruled out the easy fix. The carrier feeds arrived late on some days, so a fix that reprocessed the full history every night would miss the weekly close unless the job window was widened. The warehouse contract also limited how often the team could reload archived files without raising costs for the storage tier. Stakeholders had different tolerances: the director wanted a corrected dashboard within the week, account managers wanted a clear note about what had changed, and clients on contracts wanted a reconciliation they could check themselves.
- In the second weekDirector spotted a four-point gap between dashboard and carrier statement.
- Two days laterLineage rebuilt from raw carrier files to the published lane total.
- Later that weekMapping gap found: one exception code treated two different ways.
- Before the review dateReconciled figure published with a method footnote and dated change notes.
- End of the monthReconciliation checks stopped two later publishes that would have repeated the gap.
Actions taken
- Renata froze the spreadsheet adjustments and asked each owner to list their rules, which turned a hidden set of corrections into a reviewable table of fourteen entries.
- Sanjay added an explicit mapping for the delivered-with-exception code, with a test that fails if any new carrier status appears without a row in the mapping table.
- The team added row counts and a lane-level reconciliation check at each stage, so a mismatch above a set tolerance stops the publish job and notifies the data owner.
- Account managers received a dated change note with each corrected figure, and the dashboard showed a small badge when a lane had been restated in the last thirty days.
Results
The lane's published on-time rate moved to within half a point of the carrier figure after the mapping fix, and the remaining gap traced to two shipments with missing timestamps. Renata's contract review went ahead with the reconciled number and a footnote explaining the method. Of the fourteen spreadsheet rules, eleven were retired because the transformation layer now handled them, and three were moved into version-controlled configuration with named owners. Lattice has not had a restated figure reach a client since, though two lanes still need better carrier timestamps.
Questions for discussion
- Which lineage evidence should a team keep so that a published figure can be traced back to the raw records months later?
- When a spreadsheet adjustment is more accurate than the pipeline, how should the team decide whether to codify it or fix the source?
- What tolerance would you set for reconciliation checks, and who should have authority to stop a publish job?
Case Study 3
Suspicious service account activity during an overnight window
Brindle Clinic Network, a group of three outpatient clinics that share scheduling and records systems.
The situation
Brindle manages scheduling, patient messaging and a shared records gateway for three outpatient sites. Its network, identity roles and log retention rules are all declared in infrastructure code, reviewed by a small platform group and applied from a pipeline. Early in the second week, the security monitoring service flagged a service account logging in from an address range the organization had never used. The same account had made several configuration writes to the messaging gateway over the previous night. Nobody could yet say whether the writes were routine automation or something else.
The challenge
Ines, the security lead, had to decide how far to act before the picture was clear. Pulling the account's permissions would stop any real attacker, but it would also stop the overnight reminder job that sends appointment texts to about four thousand patients. Ahmed, the platform engineer who owned the pipeline, was on leave and reachable only by message. The clinic managers wanted to keep the schedule running, and the compliance officer wanted every step recorded in a form that would hold up to an outside audit.
Analysis
Security work in a clinic network is easy to overreact to, because the cost of a missed compromise is high and the cost of an outage is also high, and both land on patients. The objective was to keep appointments and messaging running while establishing whether the account had been used by someone else. The evidence came from three places: the identity provider's sign-in records, the pipeline's run history, and the gateway's configuration audit log. Those sources disagreed on timing by several minutes, so the first step was to agree on a single clock and a single list of events.
The constraints were sharper than in a typical incident. Any containment that touched the reminder job could delay care notices, and the clinics had no manual fallback for every reminder. Preserving evidence meant taking snapshots of the account's role bindings and log exports before changing anything, which took time during a period when the trail could still be altered. Stakeholders needed different things: patients needed their appointments honored, clinic managers needed a schedule they could trust, and the regulator's reviewers would later ask why each action was authorized and by whom.
Actions taken
- Ines had the platform group export the account's role bindings, sign-in history and gateway audit log to write-once storage, with hashes recorded in the incident file.
- The account was restricted to read-only on the gateway and its new sign-ins were blocked, while the reminder job was moved to a separate account with a fresh, narrowly scoped key.
- Every containment step was logged with the authorizing person, the time and the reason, and the compliance officer countersigned each entry before the next step began.
- Ahmed reviewed the pipeline's run history by message that evening, and the team agreed no key would be rotated or deleted until his findings were recorded.
Results
The review found that the overnight writes came from a scheduled job pointed at the gateway by an out-of-date configuration, and the new address range came from a contractor's remote access tool that had been enrolled without a review. Nothing in the evidence showed data had left the gateway, though the team reported the event to its regulator as a precaution. Appointment reminders went out on time through the separate account. The stale key was retired, the remote tool was removed from the approved list, and the enrollment step now requires a platform review.
Questions for discussion
- What evidence would have made the scope clear faster, and which of those sources should be retained in write-once form by default?
- When a containment step would disrupt patient-facing work, who should be allowed to approve it, and how fast must that approval happen?
- How should a team document an authorized response so that the record is useful to an outside reviewer who was not in the room?
Case Study 4
Choosing a multi-region design for event ingestion
Quillfeather Analytics, a small software company that sells usage dashboards to independent retailers.
The situation
Quillfeather collects point-of-sale events from about 1,200 small retailers and turns them into dashboards the next morning. Ingestion currently runs in one region, with a single managed queue feeding a set of workers that write to a relational store. Over the past several quarters, event volume has roughly tripled, and the evening batch now finishes close to the start of business hours. Retailers in two other regions have also begun asking for lower-latency views, and one has written a contract clause that specifies a maximum delay. The team has three candidate designs on the table, and each one has a different cost curve.
The challenge
Teodora, the platform architect, had to recommend a design before the next budget cycle. Multi-region replication would meet the latency clause but roughly double the storage and egress line. A streaming pipeline with partitioned workers would cut the batch window, yet the team had only two engineers who had run one in production. Doing nothing was not a serious option, because the contract clause would be breached within two quarters at current growth.
Analysis
The business objective was to keep the dashboards current for retailers without making the platform cost more per event than the subscription revenue supported. Those two goals pull in different directions, and the team's first move was to separate them. Evidence was reasonably good: six weeks of load logs gave the peak rate per region, the storage bill broke down by table, and a load test had already shown where the relational writes saturated. What the evidence did not show was how retailer demand for faster views would grow, so the team modeled two scenarios and labeled the guesses as guesses.
Constraints narrowed the options quickly. The security review required that retailer data stay within the region where the retailer was registered, which ruled out a single global store with cross-region reads for personal fields. The team also had a fixed platform headcount and a rule that no new service could page the on-call rotation without a runbook. Stakeholders diverged: the sales lead wanted the contract clause met in full, finance wanted a cost ceiling per retailer, and the two engineers who knew streaming wanted a design they could operate without weekend work.
- Measure lag Per-region lag and queue depth reviewed daily against the contract clause.
- Replay and reconcile Replay from the event log after any fault, then compare totals with source counts.
- Tune partitions Adjust partition counts when a region crosses its load threshold for two weeks.
- Review unit cost Compare storage and egress per retailer with subscription revenue at each planning cycle.
Actions taken
- The team built a streaming prototype for the two busiest regions, with partitions keyed by retailer, and ran it beside the batch job for six weeks without showing results to customers.
- Teodora split the work so the two experienced engineers wrote the partition design and runbooks, while two others paired on the consumer code and took on-call shadow shifts first.
- The team kept retailer records in their home region and replicated only aggregated counts across regions, which met the clause for dashboard freshness without moving personal fields.
- Finance and engineering agreed a unit cost target per active retailer, with a monthly report, and a rule that any region breaching the target for two cycles triggers a design review.
Results
The team chose a hybrid: streaming ingestion in the two busiest regions, with regional stores and aggregated cross-region replication. The contract clause was met within the first planning cycle, and the evening batch window shrank to a short cleanup job. Monthly cost per retailer rose modestly, though it stayed under the ceiling finance had set. Two regions remain on batch for now because their volume does not justify the extra on-call load, which was the outcome the headcount constraint had pointed toward.
Questions for discussion
- Which measurements would have changed the recommendation, and how early should the team have collected them?
- If the two regions left on batch began to breach their contract clauses, what trigger should move them to streaming?
- How should a team weigh a design that is cheaper to run against one that is easier to staff through an on-call rotation?
Case Study 5
Auto-approving infrastructure changes with an AI reviewer
Fernhollow Retail Systems, a mid-sized retailer that runs its own store and warehouse platforms in three countries.
The situation
Fernhollow's platform group manages around 2,400 infrastructure modules that define stores, warehouses and payment hosts. Every change is reviewed by a human before it is applied, and reviews are a known bottleneck: the median pull request waits most of a working day for a second approver. A vendor proposed an AI reviewer that reads each plan, scores its risk, and automatically approves changes below a threshold. The pitch was faster merges and fewer stalled releases, and the sales deck cited results from a pilot at another company. Nobody on the team could check those results.
The challenge
Imran, the platform manager, had to decide whether to pilot the tool and on which changes. The vendor's demo approved a neat change to a load balancer, but nobody had shown it a change to the tax-rate module, which touches every store. The reviewers who would be overridden were senior engineers whose past approvals had shaped the training history, and a tool learned from that history might quietly favor their habits. Two teams in other regions worried that low-risk labels would hide the changes that matter most to them.
Analysis
The business objective was to shorten the time between a reviewed change and a live change without weakening the controls that keep stores and payment hosts running. Speed was a real goal, since stalled releases cost the merchandising team a planning window each time. But a faster reviewer that is wrong about a payment host is worse than a slow one. The evidence was thin: a short vendor benchmark on synthetic plans, and the team's own history of two hundred and eleven reviewed changes with outcomes recorded. The first job was to define what counts as a good decision before the model was judged on it.
Operational limits made a clean experiment hard. Infrastructure plans are not independent: a small module change can alter dozens of resources, and the reviewer sees only the plan text, not the blast radius across regions. Any pilot had to run in shadow mode first, which meant reviewers would still approve everything while the tool's labels were logged. Stakeholders read the same pilot differently: merchandising saw a quicker path to launch, security saw a new approver that could not be interrogated, and the regional teams wanted assurance that their changes would never be scored by a standard tuned on someone else's estates.
Actions taken
- Imran set the evaluation as a labeled set of two hundred and eleven past plans, with each reviewer's decision and the later incident record, and scored precision on high-blast-radius changes first.
- The team defined the stop conditions in advance: any missed change that later caused an incident, or any approval on a payment or tax module without a human check, would halt the pilot.
- Reviewers stayed in the loop for every change during shadow mode, and the team audited approval rates by team and region each week to catch any systematic gap in scoring.
- Auto-approval was limited to a short list of module types and a single region at first, with a one-step rollback tested before each widening of the scope.
Results
Shadow mode ran for the planned period and found that the tool agreed with reviewers on most narrow changes but missed two plans that expanded a shared tax rate table. Both were caught by the human reviewers still in the loop. Imran did not roll out auto-approval to payment or tax modules at all. Auto-approval now covers tag and log retention changes in one region, with a weekly audit of approval rates. The vendor was asked to publish its scoring method, and the platform group keeps the final decision on any change to a shared module.
Questions for discussion
- What would a fair scoring method look like for changes proposed by engineers with very different review histories?
- Which stop condition would you add, and how would you test that it fires before a change reaches production?
- If the vendor refuses to explain its scoring, what minimum evidence should the team require before widening any automated approval?
Learning Resources
Technical CASE STUDIES
China, India, United States, Indonesia, Pakistan, Nigeria, Brazil, Bangladesh, Russia, Japan, Mexico, Philippines, Vietnam, Egypt, Turkey, Germany, United Kingdom, France, Italy, Canada, South Korea, Australia, Spain.