Cloud Architecture Planning Case Studies
Technical5 Case StudiesUpdated Sep 2026
The five cases below each begin with a team under pressure: a release that slowed checkout, two reports that disagreed, an archive read pattern nobody could explain, a storefront that outgrew its design, and a vendor tool that promised to clear a backlog. The organizations are fictional, but the trade-offs will be familiar to anyone who has been paged at night or asked to justify a hosting bill. All five are read against the same four concerns: system requirements, performance evidence, access controls, and maintainability. Read them as working notes from people who chose under pressure, not as answer keys.
Case Study 1
Checkout latency after a Thursday release
Brightwater Clinics, a network of eleven outpatient clinics running its own scheduling platform with a platform team of nine engineers.
The situation
Brightwater's patient scheduling service had been stable for most of the year. On a Thursday afternoon, the team shipped a release that changed how appointment slots were cached. Within the first hour, median response time on the booking endpoint rose from about 180 milliseconds to nearly four seconds, and the error rate climbed past eight percent during the afternoon peak. Front desk staff at three clinics reported that patients were being turned away at the kiosk. The on-call engineer, Priyesh, rolled back the release, but latency stayed high for another forty minutes. The service ran on eighteen replicas behind a load balancer, and the database connection pool was sized for the old traffic pattern.
The challenge
The rollback did not restore service, which meant the release was not the only cause. Priyesh had read access to the metrics dashboard but not to the database console, and the platform lead, Ruth, was on a flight with no connectivity. Two engineers wanted to scale the replicas upward, while another wanted to disable the cache entirely. Each change would have been reversible in theory, but nobody could say which one would help without evidence.
Analysis
The first question was what the system had to do under load, and whether the release had changed that. The booking service had a stated target of responding within half a second at the ninetieth percentile, and the release notes did not mention it. Performance evidence came from three sources: application traces, database wait events, and load balancer logs. Traces showed time spent waiting for connections, not computing, which pointed toward pool exhaustion rather than the cache. Teams grounded in Cloud Architecture Planning usually treat the first hour of an incident as evidence gathering rather than repair, because each reversal erases some of the signal needed to explain the failure.
Access controls shaped the response too. Priyesh could see metrics but could not inspect active sessions or change pool settings, so the fix had to pass through someone with console rights. The team had granted that right to two people, and only one was reachable. A break-glass procedure existed on paper but had never been tested. Maintainability was the deeper problem: the pool size sat in a configuration file, and nobody had documented why it was set to that value. Any preventive measure had to make those settings visible, reviewed, and tied to the load they were meant to serve.
Actions taken
- Reinstated the previous connection pool size through the break-glass role, with a second engineer watching the change log, and confirmed latency fell within ten minutes.
- Froze further deployments, opened a shared incident timeline, and asked the front desk leads to log each turned-away patient so that the scope of harm could be measured later.
- Added pool utilization and wait time to the on-call dashboard, and moved the pool size into a reviewed configuration file with a comment linking it to the expected peak concurrency.
- Wrote a runbook for the break-glass role, tested it in staging during the following week, and required two named approvers for any change to database connection limits.
Results
Service returned to baseline latency within the first hour after the pool change, and the error rate fell below one percent by the end of the afternoon. The front desk logs recorded 212 turned-away bookings, which the clinic managers used to contact patients directly. The post-incident review found that the release itself was mostly innocent: the cache change added a lookup that was harmless under normal load but multiplied connection requests when slots were refreshed in bulk. The team kept the cache change with a corrected refresh interval. Break-glass access was tested twice in the following month, and both drills took under fifteen minutes.
Questions for discussion
- Which settings in your services were set once and never explained, and who would notice if they drifted?
- If the only person with console rights were unreachable for an hour, what could your team actually change?
- How would you choose between a rollback and a targeted fix when the evidence points in two directions?
Case Study 2
Conflicting margin reports from unreconciled carrier invoices
Quillmark Logistics, a regional freight broker with about 60 staff and a finance team that reports weekly margin figures to its board.
The situation
Quillmark's pricing team noticed that two weekly reports disagreed about lane margins. The operations dashboard showed a 6 percent margin on refrigerated freight in the Gulf region, while the finance workbook showed 2 percent for the same period. Both were built from the same warehouse, but they drew from different ingestion jobs. The warehouse received carrier invoices from nine partners in different formats, and about 4 percent of rows arrived with missing fuel surcharges or currency codes. Nobody had a record of which transformation had dropped or duplicated those rows. A pricing manager had already adjusted two contracts based on the higher figure.
The challenge
The disagreement was not simply a bug to fix. Each team had a plausible definition of margin, and the warehouse had no lineage tables linking a reported number back to source files. Dates were also inconsistent: some carriers invoiced by pickup date and others by delivery date, so a shipment straddling month end could land in either period. The data engineering lead, Marisol, had three weeks before the next board pack and no authority to freeze either report.
Analysis
The first requirement was agreement on what the number meant. Margin had been defined in two places with different treatments of accessorials and surcharges, and neither definition appeared in the data dictionary. Performance evidence here meant reconciliation evidence: row counts, control totals, and sample tracing from each report back to invoice lines. Marisol ran the same query along both paths and found roughly 1,400 invoice lines present in one ingestion job and absent from the other. Reconciling counts at each stage before debating formulas is usually the faster route, since a disagreement in totals tends to point to a join or filter rather than a difference in philosophy.
Access controls mattered because the transformation logic lived in a shared scheduler that three teams could edit, and the change history did not show who had altered the currency conversion step. Finance had write access to the reporting schema, which made it easy to patch a figure by hand and harder to trace later. Maintainability suffered from the same gap: the mapping from carrier formats to the canonical invoice model existed as nine undocumented scripts. Any repair had to separate read access for analysts from write access for pipeline owners, and it had to put each carrier mapping under version control with a named owner.
- Trace lineage Follow each reported figure back to source rows and record exclusions.
- Validate transforms Compare counts and sums at every stage against carrier statements.
- Monitor variance Alert when daily totals drift beyond the agreed tolerance.
- Restrict and version Limit write access and keep every mapping under version control.
Actions taken
- Traced the twelve reported figures back to source invoice lines using the ingestion job logs, and recorded each join or filter that excluded rows in a shared lineage sheet.
- Agreed a single margin definition with finance and pricing, including how accessorials, fuel surcharges, and currency conversion were treated, and published it in the data dictionary with an owner's name.
- Added control totals at each pipeline stage, comparing invoice counts and summed amounts against carrier statements, and set alerts when a carrier's daily variance crossed two percent.
- Moved the nine carrier mappings into version-controlled files with named owners, and restricted write access to the reporting schema to the pipeline service account.
Results
The lineage work found that a date filter was excluding late-month deliveries from one carrier, and a currency step was applying the wrong rate to Canadian invoices. Corrected, the Gulf refrigerated margin settled at 3.4 percent, which sat between the two earlier numbers and matched neither. Pricing reversed one of the two contract changes and kept the other after review. The board pack went out on schedule with a reconciliation appendix. Within the next two reporting cycles, variance alerts caught a changed file layout from one carrier before it reached any report.
Questions for discussion
- When two reports disagree, which team would you ask first to define the number, and what evidence would they need to bring?
- Where in your own pipelines could a filter remove rows without anyone recording it?
- How would you separate the people allowed to correct a figure from the people allowed to change how it is calculated?
Case Study 3
Unusual archive reads from a service account
Caldera Health Exchange, a nonprofit that routes lab results between about 300 member clinics and their ordering physicians.
The situation
Caldera's security analyst, Teodor, noticed an unusual pattern in the authentication logs on a Tuesday evening. A service account used by the lab results integration had authenticated from an address range it had never used, and it had requested a list of every practice identifier in the directory. The account belonged to a reporting job that ran nightly from a fixed set of hosts. Over the next four hours, the same account made about 2,300 reads against a results archive, mostly in alphabetical order by patient surname. The integration team could not confirm whether any of the calls were theirs, and the on-call rotation was thin that week.
The challenge
The team had to answer three questions without yet knowing the first: was the account compromised, how far had access spread, and what could be shut off without stopping lab results that clinics depended on every morning. Preserving evidence conflicted with the impulse to rotate credentials immediately. Rotating first would lock out legitimate calls and erase the session tokens that showed where the requests came from.
Analysis
The system requirement that mattered most was the ability to answer what was accessed, by whom, and when, for every read of protected health data. Caldera's logs met that requirement at the API gateway but not inside the archive, which recorded only aggregate query counts. Performance evidence came from gateway logs and network flow records, which showed source addresses, request rates, and byte volume per hour. The requests were steady rather than bursty, which suggested scripted access instead of a person browsing. Scoping the incident meant building a timeline from these sources before deciding what counted as affected, and recording which conclusions were inferred and which confirmed.
The access control failure was ordinary and therefore common. The service account held read permission on the whole archive, not the three tables its job needed, and its credential had gone unrotated for a long time because rotation broke a scheduled job that nobody owned. Maintainability shaped the response because the same permission model appeared in four other integrations. Containment could not be a single emergency change. It had to narrow the account's scope, move it behind a short-lived credential, and document the authorized steps so the next responder could follow them without improvising. Each step needed an owner and a rollback path written before it was executed.
Actions taken
- Preserved gateway logs, flow records, and the account's session history in write-once storage before changing any credential, and recorded the hash of each export.
- Narrowed the service account to the three tables the nightly job needed, and blocked the unfamiliar address range at the gateway, with the integration lead approving each rule in writing.
- Issued a new short-lived credential to the nightly job, scheduled during a window agreed with clinic operations, and rotated the old secret only after the job succeeded.
- Wrote the containment steps as an authorized runbook with named approvers, and notified the privacy officer before any finding about affected patients was shared outside the response team.
Results
The timeline showed that the account had made 2,300 reads over the four-hour window, all from the unfamiliar range, and that roughly a third of them touched surnames in one letter range before the block took effect. The privacy officer determined which patient records were affected and arranged notices through the formal process. The integration team later confirmed that the account had been used by a leaked script on a contractor's laptop, which had been removed from the network. Containment held without interrupting morning lab deliveries. The review also led to scoping the other four integrations within the following month.
Questions for discussion
- What evidence would your team lose first if credentials were rotated before logs were preserved, and how would you know?
- Which service accounts in your environment hold more access than their jobs need, and who would have to agree to narrow them?
- Who should decide that a suspected compromise is serious enough to interrupt a clinical or production workflow?
Case Study 4
Choosing how to scale a growing storefront
Tessellate Retail, an online seller of household goods with about 45 engineers and a catalog of 80,000 products.
The situation
Tessellate's storefront had grown from one regional market to six countries in two expansion phases. The original design ran the product catalog, pricing, and inventory inside one service backed by a single relational database. At peak promotional hours the database handled about 9,000 queries per second, and listing pages had crept above the two-second target that the marketing team had set. Inventory reads made up most of that load, and most of them tolerated data up to thirty seconds old. Finance wanted hosting costs to stop rising at the same pace as order volume, which had roughly doubled over the past several months.
The challenge
The platform lead, Benedikt, had three options on the table: scale the database vertically, split the monolith into services along domain lines, or add a read-optimized cache and replica layer in front of the existing system. Each promised relief for latency, but they differed sharply in cost, operational load, and the security boundary they would create. The team had nine engineers who understood the existing service and only two who had run a distributed data store in production.
Analysis
The workload profile carried most of the weight in this decision. Reads outnumbered writes by roughly forty to one, and inventory reads could accept staleness while checkout writes could not. Performance evidence came from a four-week load replay against staging, which showed that connection handling, not CPU, was the first bottleneck under promotional load. That finding weakened the case for simply buying a larger instance. A replica layer would absorb most read traffic, but only if the team could bound how stale inventory data became while still honoring checkout guarantees. Requirements had to be written as numbers before any option could be scored fairly.
Access controls and maintainability pulled the options in different directions. A service split would create new network boundaries that needed service identities, secret rotation, and audit trails for each caller, which the current team could not yet staff. A replica layer kept the existing permission model but added a second path to customer data that had to be covered by the same policies. Benedikt's team favored the option whose failure modes they could explain to the on-call rotation within a single afternoon, even though it was not the cheapest on paper. Maintainability was treated as a constraint on design rather than a cleanup task to schedule later.
Actions taken
- Ran the load replay again with the proposed replica and cache layer, measuring connection counts, staleness of inventory reads, and checkout latency at the ninetieth percentile.
- Defined an inventory staleness budget of fifteen seconds for listing pages, with checkout reading directly from the primary database and a separate reservation step for stock.
- Gave the cache layer its own service identity with read-only grants on the catalog schema, and added it to the quarterly access review alongside the primary database credentials.
- Chose the replica layer for the first phase and set a written trigger, based on sustained connection saturation, that would reopen the question of splitting the catalog service.
Results
Under the next promotional period, listing latency stayed below the two-second target at the ninetieth percentile, and database connections peaked at about a third of the earlier level. Hosting costs rose by roughly eight percent while order volume grew about thirty percent, which finance accepted. Two inventory staleness complaints were traced to a cache warm-up bug, fixed before the next event. The split-service question stayed open. Connection saturation reached the written trigger in the following quarter, so the team began planning the catalog extraction with its access model already defined.
Questions for discussion
- Which of your current latency or cost problems would a read replica actually fix, and how would you find out before building it?
- What staleness budget would your customers accept on a product page, and who is responsible for writing that number down?
- If you added a second path to sensitive data, what would have to be true for the on-call team to explain its failure modes within an afternoon?
Case Study 5
Evaluating an automated underwriting document reader
Northfield Credit Union, a member-owned lender with about 120 staff and an 18-person underwriting team.
The situation
Northfield's small-business loan queue had grown to about 3,400 applications awaiting review, and average time to decision had stretched past eleven business days. A vendor proposed a document reader that would extract income figures, flag missing statements, and draft a first-pass risk note for each application. The pitch promised a sharp drop in review time and fewer data entry errors. Underwriters had seen similar demonstrations before. The vendor's accuracy figures came from a benchmark built on documents that looked nothing like Northfield's, with many scanned statements from regional banks the credit union had never worked with.
The challenge
The credit union had to decide whether to pilot the tool at all, and if so, how to measure it. A model that quietly misread one kind of document could produce faster decisions that were systematically wrong, and fair lending obligations meant that a skew by applicant group would be a serious problem, not a minor one. The compliance lead, Yusuf, wanted evidence the team could defend to examiners, but the pilot had to start within the quarter because the queue was already pushing members toward other lenders.
Analysis
Requirements came first, and they were stricter than the vendor's brief. The tool had to extract the same income and debt fields a trained underwriter would, with each value traceable to a page and line in the source document. Evaluation metrics needed to go beyond overall accuracy. The team defined field-level precision and recall on a held-out set of 1,200 applications drawn from recent real submissions, stratified by document type and by the bank that issued each statement. Performance evidence also tracked how often the draft risk note changed a decision that a human had already made correctly, since a helpful-looking draft can pull reviewers toward its errors.
Access controls had to answer who could see member data, who could send it to the vendor, and who could change the prompt or extraction rules. The vendor's hosted version would receive full applications, which meant a data processing agreement and a review of retention terms before any file left the credit union's environment. Inside that environment, the tool received read access to the document store and no write permission on loan records. Maintainability was the quieter concern. A model whose behavior shifted with each vendor update would invalidate the evaluation unless the credit union pinned a version and re-ran the held-out set before accepting any change.
- 1Shadow evaluationCompare extracted fields with final underwriter entries on held-out files.
- 2Subgroup reviewBreak error rates down by document type, issuing bank, and applicant group.
- 3Limited pilotStandard business loans only, with human confirmation before scoring.
- 4Automatic stopHalt the pilot if any document group exceeds the agreed error tolerance.
- 5Monthly compliance reportReport subgroup outcomes and version changes to compliance each month.
Actions taken
- Ran the tool in shadow mode on 1,200 applications, comparing extracted fields and draft notes with the underwriter's final entries, and broke the results down by document type and issuing bank.
- Set a rule that no automated value reaches a decision without a human check: every extracted income figure links to its source page, and underwriters confirm or correct it before scoring.
- Pinned the vendor model version, required a full re-run of the held-out set before any update, and kept the vendor's access limited to a dedicated environment with no retention of files.
- Set rollout gates: pilot only standard business loans, stop automatically if any document group's field error rate exceeds the agreed tolerance, and report subgroup outcomes to compliance monthly.
Results
Shadow results showed field-level accuracy above target on typed statements but a clear drop on scanned statements from two regional banks, where income figures were misread in about one case in nine. The team narrowed the pilot to typed documents and kept the scanned group in manual review. Underwriters found the draft notes useful on straightforward files, mostly as a checklist. Time to decision fell by roughly three days for the pilot group, and the compliance review found no disparity in error rates across the applicant groups tested. Expansion was deferred until the scanned-document gap was addressed.
Questions for discussion
- What error rate in one document group would be enough to stop an automated tool, even if the overall average looked good?
- Who is responsible for re-running an evaluation after a vendor changes a model, and is that person named in the contract?
- When a helpful draft and a careful reviewer disagree, how would you measure which one was usually right?
Learning Resources
Technical CASE STUDIES
China, India, United States, Indonesia, Pakistan, Nigeria, Brazil, Bangladesh, Russia, Japan, Mexico, Philippines, Vietnam, Egypt, Turkey, Germany, United Kingdom, France, Italy, Canada, South Korea, Australia, Spain.