Software Architecture and Engineering Best Practices Case Studies
Technical5 Case StudiesUpdated Sep 2026
Architecture and engineering practice rarely fail in one dramatic place. They fail in release notes nobody read, dashboards that drifted, and permissions granted for a week that stayed for a year. The five cases below come from product and platform teams. Each starts with a concrete problem: a degraded service, reports that disagree, a suspected intrusion, a scaling bottleneck, or an automated tool that quietly made mistakes. The same habits decide the outcome: stating requirements before debating designs, reading performance evidence before changing code, treating access as part of the design, and leaving systems the next engineer can change.
Case Study 1
Checkout slowdown after an evening release
Brightwater Ticketing, a regional event ticketing company with about 40 engineers and one primary checkout service.
The situation
Brightwater Ticketing sells seats for concerts and sports events through a web checkout that depends on a pricing service, a payment gateway adapter, and a shared session store. The evening before a major on-sale, the pricing team shipped a release that changed how cart totals were cached. For roughly forty minutes after the deployment, p95 checkout latency climbed from under 800 milliseconds to over 6 seconds, and error rates rose on the payment confirmation step. The on-call engineer, Priya, rolled back the pricing service, but the slowdown continued. Traffic was normal, so the cause was not a spike. Support began receiving complaints about duplicate charges and stalled carts.
The challenge
The rollback did not restore the old behavior, which pointed to something else in the release or in shared state. The team had 30 hours before the on-sale. Dana, the engineering manager, needed to decide whether to freeze the checkout path, reroute traffic to a queue, or keep investigating live. Each option had a cost. The system had no dashboards that separated cache hits from database reads, so the evidence was thin.
Analysis
The first step was writing down what checkout had to do under load. The product requirement said confirmation within three seconds at the 95th percentile during on-sale peaks, and that number became the test for every hypothesis. Traces showed the time was spent waiting on a lock in the session store, not in pricing computation. Cache hit ratios had not moved, which ruled out the obvious explanation. The evidence pointed to contention created by the new cache write pattern, which wrote the full cart on every price read. Before touching the code again, the team tied the trace to that one component and wrote the threshold into the alert.
The incident also exposed an access problem. The pricing service's configuration, including the cache time to live, could be changed by any engineer with deploy rights to the shared platform repository, and the change had gone through without a second reviewer because the pipeline treated configuration as documentation. Restricting configuration writes to the owning team, with a required reviewer from the team that consumes the values, closed that gap. On maintainability, the cache logic had three undocumented callers. The fix was not clever. It was a short module boundary, a written contract for what each caller may assume, and a test that fails when the contract changes.
- Evening of releaseCart totals start caching on every price read.
- Twenty minutes laterLatency passes six seconds, and rolling back pricing does not help.
- Forty minutes inTraces point to lock contention in the session store.
- Flag restores old patternTraffic returns in steps of ten percent while latency is watched.
- Next morningAlerts tied to the three-second requirement and config ownership changed.
Actions taken
- Restored checkout within the window by switching the cart cache back to the previous write pattern behind a feature flag, then reenabling traffic in steps of ten percent while watching the 95th percentile latency.
- Added spans around the session store lock and the cart write path, with alert thresholds tied to the three-second requirement rather than to average latency, so the next slowdown would page before customers noticed.
- Changed configuration ownership so cache settings live in the pricing repository, require review from the checkout team, and cannot be edited through the shared platform pipeline without that approval.
- Wrote a contract document for the cache module listing each caller's assumptions, and added a contract test that fails the build when a caller's expected fields or timing change.
Results
Checkout returned to normal latency within the window, and the on-sale ran without a second incident. Confirmation times at the 95th percentile settled below two seconds at peak. Duplicate charges were traced to client retries that fired after timeouts, and a request key was added to make retries idempotent, which reduced the duplicate reports to a handful that were refunded. Later, the team's alert fired twice on unrelated database slowdowns, and both were resolved before customers reported anything. The review board noted that the cache change had passed unit tests but never ran under concurrent cart writes.
Questions for discussion
- Which of the three numbers in the requirement would you have chosen to alert on, and why would average latency have hidden this incident?
- If the cache settings had belonged to the checkout team from the start, which review and rollback steps would have changed during the release?
- How would you decide whether a rollback is the right first move when the rollback itself has not restored the previous behavior?
Case Study 2
Two spoilage reports and no shared definition
Hollin Fresh Supply, a regional produce and dairy distributor serving about 220 grocery stores.
The situation
Hollin Fresh Supply distributes produce and dairy to about 220 grocery stores from three regional warehouses. Two weekly reports now disagree about spoilage. Operations reports a spoilage rate of 4.1 percent for the most recent month, while finance reports 2.7 percent from the same month. Operations reads scanner events directly, while finance uses a nightly summary that joins those events with returns and a manually maintained adjustments sheet. Nobody had documented which join dropped which rows, and the gap had widened as two new stores were onboarded with different store codes.
The challenge
The analytics lead, Marcus, had to produce one number for the regional review in less than two weeks, and he could not simply pick the higher or lower figure. Procurement was already reducing orders based on the lower rate, while operations had changed staffing for the higher one. The team needed to know which transformation step introduced the difference, whether the adjustments sheet had its own errors, and what monitoring would catch the next divergence before a meeting did.
Analysis
The team began by writing down what spoilage was supposed to mean, since the two reports used different denominators. Operations counted units received at store docks, while finance divided by units shipped from the warehouse. That was a requirements problem before it was a data problem. Once both definitions were fixed in writing, the team ran row counts at each stage of the nightly summary. The join to returns dropped about 3 percent of scanner events with null store codes, mostly from two newly onboarded stores that had the highest spoilage. The drop raised no error and no warning in the job log.
The adjustments sheet was edited by four people through a shared link, with no change history and no owner. Any figure in it could move the finance number without anyone seeing the change. Moving adjustments into a governed table with named editors, timestamps, and a reason field addressed the access question. The store code mapping was hard-coded inside the nightly job, which meant that every new store required a code change by someone who did not own the data. Teams working in Software Architecture and Engineering Best Practices usually move that mapping into a reference table with its own validation, so that onboarding is a data change and the join logic stays stable.
Actions taken
- Built a row-count and value-total check at each stage of the nightly summary, failing the job and paging the analytics on-call when dropped rows exceed one percent of scanner events.
- Moved the adjustments sheet into a governed reference table with named editors, a reason field for every change, and a history view that finance and operations can both read.
- Replaced the hard-coded store mapping with a versioned table that carries open and close dates, so new stores enter through a data change reviewed by the owner of store records.
- Wrote a one-page metric definition for spoilage that names the denominator, the source table, and the owner, and linked it from both dashboards so the next disagreement starts from the same page.
Results
The regional review used a single spoilage figure of 3.4 percent, with the definition printed in the footer and the reconciliation attached as an appendix. The remaining gap between the two earlier numbers was explained almost entirely by the dropped rows and by a double-counted return batch in the adjustments sheet. Procurement reversed part of its order reduction within a week, which cost little, and operations kept its staffing plan until the next quarter. Over the following quarter, the row-count check caught one partial load from a warehouse feed, and the job stopped before any dashboard refreshed.
Questions for discussion
- Which of the spoilage definitions in your own organization would you write down first, and who would own the disagreement if two teams refused the wording?
- When a join silently removes rows, what threshold would you set before the job fails, and how would you justify it to the team that depends on the output?
- Where should store reference data live when both operations and finance need to change it, and what access model keeps each team from editing the other's inputs?
Case Study 3
Unusual token use on the freight portal
Larkfield Logistics, a mid-sized freight broker with a customer portal and about 65 staff.
The situation
Larkfield Logistics runs a customer portal where shippers book loads and check invoices, backed by a rate-quoting API and an object store for shipping documents. In the second week of the quarter, the monitoring team noticed a service account making requests from an unfamiliar cloud region, at hours when the account was normally idle. The account belonged to an integration used by a small carrier partner. Access logs showed about 2,300 document downloads over six hours, which was roughly forty times the account's normal volume. No one could yet say whether the partner's credentials had leaked, whether the integration had a bug, or whether the activity was an insider using the account.
The challenge
The security lead, Tomasz, had to decide how far to act before the picture was clear. Disabling the account would stop the downloads but could interrupt load bookings for the partner during a busy shipping day, and it would alert whoever was using the credentials. Leaving it running risked exposure of customer bills of lading. Any step taken also had to be recorded so that the carrier partner, the company's cyber insurer, and internal auditors could later see who authorized what, and when.
Analysis
Scoping started from requirements already written down. The portal had to stay available for bookings during shipping hours, and shipping documents had to be retrievable only by the account holder or an authorized employee. Those two requirements pulled in opposite directions, so the team needed evidence before choosing a step. They pulled the object store access logs, the identity provider sign-in records, and the rate API call history for the same six hours, and compared the download rate against the account's baseline over the previous ninety days. The rate was abnormal, but the timing and source addresses alone did not show whether a human or an automated job was responsible.
The access design was the real weakness. The integration account held a long-lived key with read rights to every shipping document in the bucket, not only the partner's own loads, and the key had not been rotated since the integration was built. Containment therefore meant narrowing the account to the partner's prefix and issuing a new short-lived credential, which kept bookings running while cutting the exposed surface. Before any change, the team exported and hashed the logs and asked the carrier to confirm which of its staff held the old key. Maintainability needed a named owner and a rotation schedule a new engineer could follow.
- Scope affected data Identify accounts, objects, and time window from access logs.
- Preserve evidence Hash and hold logs before any access change is made.
- Contain exposure Narrow scope and rotate credentials under named approval.
- Document steps Record each action, approver, and outcome in the incident file.
Actions taken
- Revoked the integration's long-lived key and issued a short-lived credential scoped to the partner's document prefix, after logs were exported and hashed for preservation.
- Put the access logs, audit records, and sign-in history under a hold for the prior ninety days, and recorded each export with its hash, time, and operator.
- Asked the carrier partner, through its named contact, to list staff who had access to the old key, and to confirm in writing whether any of their systems had been changed in that window.
- Wrote the response steps into a runbook with named authorizers for each containment action, and filed it with the incident record so that each step can be traced to the person who approved it.
Results
Scoping found that the download pattern came from an automated sync the partner had configured to retry failed pulls without limit, and that a former contractor's copy of the key had been stored in a shared spreadsheet. No evidence showed that customer records left outside the partner's own loads, though the team could not rule out copies made before the rotation. The partner rotated its staff access within a day. Bookings stayed available. The incident record lists every containment action with its approver, and the access review found four other service accounts with broader scope than their owners intended.
Questions for discussion
- Which of the containment steps would you have needed to approve in advance, and which could a responder take on their own authority without waiting?
- If the partner's staff could not confirm who held the old key, how would that change the evidence you report to the insurer?
- What would a service account owner need to review every quarter to catch a scope like this before anyone else found it?
Case Study 4
Choosing a read path for field inspection data
Corvid Field Systems, a maker of inspection software for utility crews with 310 enterprise accounts.
The situation
Corvid Field Systems sells inspection software to utility crews in 310 enterprise accounts. Inspectors record work on a mobile app that works offline and syncs when vehicles return to depots. The product stores inspection history in one relational database that also serves customer reporting. As the accounts grew, reporting queries began competing with mobile syncs for connections, and the largest customer's nightly export now runs past the overnight window it agreed to. Sync latency during depot returns has climbed, and two outages in the last quarter traced back to connection pool exhaustion during the export. The primary database instance now costs about 38,000 dollars a month, and that line item has grown faster than revenue.
The challenge
The platform team had three options with different costs and risks: split reporting onto a read replica, move inspection history to a separate store with event-based sync, or add a caching layer in front of the mobile API. Each met some requirements. The team had to compare them against the workload profile, the data isolation rules customers required, and what a three-person on-call rotation could realistically operate.
Analysis
The comparison started with the workload. Logs showed that inspectors wrote about 85 percent of their traffic in a two-hour window when vehicles returned, and that the nightly export read most of the inspection table. Sync latency at the 95th percentile was dominated by database connection waits during that window, not by query execution time. A read replica would take the export off the primary, but replication lag during the depot burst could return stale inspection status to a crew that had just submitted a correction. The team measured lag under a replayed burst from the previous quarter before accepting the trade-off.
The security requirements eliminated the caching option faster than the others. Customers required that inspection data stay isolated by tenant, and a shared cache in front of the mobile API would need tenant-scoped keys, invalidation rules, and an audit trail for every eviction, which the current team had never operated. A separate history store would isolate tenants, but it would add a second system with its own credentials, backups, and an event pipeline that three engineers would be paged for at night. The replica kept existing access controls and backup policy in place, and it needed only one new monitoring rule. The team chose it, with a written lag budget.
Actions taken
- Moved the nightly export and all reporting queries to a read replica with its own connection pool, and capped export concurrency so the depot burst keeps priority on the primary.
- Set a replication lag budget of five seconds at the 95th percentile, with an alert tied to mobile sync status so that a crew is not shown a stale status after submitting a correction.
- Routed corrections written during the depot window to the primary, so a crew's own submission always reads back from the source of truth regardless of replica lag.
- Wrote an architecture decision record listing the three options, measured workload figures, the isolation rules, and the conditions that would trigger a separate history store.
Results
Export runs moved off the primary and finished inside the customer's agreed window for the next several export cycles. Sync latency at the 95th percentile during depot returns fell, and connection exhaustion stopped appearing in the on-call log. Replica lag stayed within budget in most bursts, but two bursts crossed the threshold, and corrections were read from the primary as designed. Infrastructure spend rose modestly with the added replica instance but remained below the projected cost of the separate store. The team has a defined trigger for revisiting the history store, which is the export runtime on the largest tenant.
Questions for discussion
- Which workload measurement would have changed your choice most, and how would you collect it before the next release rather than after?
- What lag budget would you accept for a field crew's own corrections, and what would you do if the budget could only be met by the more expensive option?
- Under what conditions would you accept a shared store with tenant-scoped keys, and what would an auditor need to see to approve it?
Case Study 5
Automated claim triage before a pilot expansion
Keldane Benefit Administration, a third-party administrator for employer health plans with about 90 client companies.
The situation
Keldane Benefit Administration receives roughly 4,000 claims a week and routes each one to fast-track approval, standard adjuster review, or a fraud review queue. Average handling time for standard review has grown, and a vendor offered a triage model that it said would cut time for fast-track decisions by about 40 percent. The vendor's accuracy figures came from a claims sample with a different plan mix. Operations ran the model in shadow mode for about 6,500 claims over five weeks, comparing its routing with the adjusters' decisions. Volume looked promising, but the fraud queue received more claims from two postal areas, and nobody had checked whether that pattern reflected real fraud or differences in how claims were submitted.
The challenge
The operations director, Ruth, had to decide whether to expand the model to every queue in the next cycle. She could not evaluate it on the vendor's numbers, and she could not treat a shadow pilot as proof of safety. The team needed metrics that measured wrong decisions, not only speed, a way for adjusters to stay in control of contested claims, and a rollout plan that could be reversed without a backlog. Auditors from two client companies had asked how decisions made by the model would be explained.
Analysis
The requirements had to be written as error measures before anyone looked at speed again. The team defined four: fast-track precision, meaning the share of auto-approved claims that a senior adjuster would also approve on audit; fraud queue precision; reviewer override rate; and handling time reported separately for each postal area and plan type. The shadow data alone could not answer the precision question, so a senior adjuster audited a random sample of 400 fast-tracked claims. The audit found nine approvals that needed review, and most involved claims with coordinated billing from one provider. That was a finding about the model's blind spot, not about the vendor's marketing.
Human oversight depended on access design. Adjusters needed to see the model's routing reason for contested claims, but the vendor's dashboard exposed raw risk scores to the whole claims floor, which invited use of scores as grounds for denial. Keldane restricted score visibility to the fraud queue and senior adjusters, and logged every override with the model version in effect. Threshold changes became a change-controlled setting with two named approvers, not a vendor configuration screen. For maintainability, the team required that the routing logic be versioned alongside its evaluation set, so that a future engineer could rerun the audit and check whether a change narrowed the postal area gap.
Actions taken
- Set a rollback rule that returns a queue to the rules engine automatically if the audited precision drops below the agreed floor or any postal area's handling time exceeds its ceiling for two days.
- Rolled out to one client's standard queue first, with a fixed share of fast-tracked claims sampled for adjuster audit, and expanded only after two consecutive audit cycles met the precision floor.
- Required that any denial based on the model's output be confirmed by a named adjuster who can see the routing reason and the claim documents, with the confirmation recorded against the claim.
- Added a fairness check to the weekly report that compares routing rates and override rates across postal areas and plan types, and opened a review whenever a gap exceeds the set tolerance.
Results
Keldane did not expand the model right away. It ran a second, narrower pilot on one client's standard queue, where the audit found that the postal area gap came mostly from a documentation field that the vendor had populated differently for two intake channels. After the team corrected the field mapping and retrained the thresholds under change control, fast-track precision met the floor in two consecutive audit cycles. Handling time improved by a fifth for that queue, which was less than the vendor had promised but enough to justify a phased expansion. The rollback rule was never triggered.
Questions for discussion
- Which of the four metrics would you accept as the single go or no-go signal, and what would you need to see before trusting the other three?
- If the postal area gap traced back to real differences in fraud rates, what would fair handling of those claims look like inside the model's routing?
- Who should hold the authority to approve a threshold change, and how would you keep that authority separate from the people who benefit from faster routing?
Learning Resources
Technical CASE STUDIES
China, India, United States, Indonesia, Pakistan, Nigeria, Brazil, Bangladesh, Russia, Japan, Mexico, Philippines, Vietnam, Egypt, Turkey, Germany, United Kingdom, France, Italy, Canada, South Korea, Australia, Spain.