Systems and Network Administration Case Studies
Technical5 Case StudiesUpdated Sep 2026
Systems and network administration sits where operations, security, and business commitments meet, so decisions that look technical often carry financial and reputational weight. The five cases below come from different sectors and team sizes. Each puts a practitioner in front of a problem without a clean answer: a degraded service during peak use, reports that disagree, a possible intrusion, a platform that must grow without breaking, and an automation proposal that promises speed while adding new failure modes. Watch what each team knew at every decision point and who paid for waiting.
Case Study 1
Restoring bid submissions after a release regression
Tallgrass Freight Exchange, a regional marketplace that connects about 1,400 carriers with shippers across six states.
The situation
Tallgrass runs a freight exchange where carriers submit bids on shipping lanes during scheduled windows. Late in the afternoon following a release to the bidding service, the operations desk began reporting that submissions were timing out for carriers in two regions. The platform team's dashboards showed median response time climbing from under half a second to more than six seconds, and error rates rose in step. The release had touched connection pooling and a serialization library, both changed in the same window. Traffic was ordinary, not a spike, which made the timing suspicious. By the time the first carrier complaint reached the account team, roughly one in five bid attempts had failed.
The challenge
The release could not be rolled back cleanly because a schema migration had shipped with it, and the migration was not reversible without a maintenance window. The change freeze ahead of a quarter-end settlement run meant the on-call engineers had about ninety minutes to act before anyone outside the team would be asked for approval. Carriers who could not bid would lose lane allocations for the cycle, and the finance group needed the exchange stable for settlement.
Analysis
The first decision was what the team was protecting. Bid submission throughput mattered more than the cleanliness of the fix, since each failed attempt cost a carrier a lane allocation that would not return this cycle. That reframed the evidence. The timing pattern pointed at the connection pool before the serialization change, and the pool metrics showed exhaustion soon after each deploy. Engineers trained in Systems and Network Administration usually start by separating what the logs prove from what the dashboards suggest, and here the two agreed only after someone compared request traces from before and after the release. Operations could live with slow confirmations but not silent drops.
The rollback question was less simple than it looked. The migration had added a nullable column and backfilled it, so reverting application code would be safe, but the pool settings had been changed in the same deploy and were read at startup. The team chose a configuration rollback that reverted pool sizes without touching schema, the only change verifiable in the time available. Evidence was thin: one replay environment had not reproduced the exhaustion under synthetic load, so the team accepted a partial fix and watched carrier-facing error rates through the afternoon. Prevention came later, through a canary stage that drives pool metrics into the release gate.
- Detect and triage Correlate carrier timeouts with the release timestamp before anyone touches configuration.
- Contain and restore Revert pool sizes in place, keep the migration, and verify against live bid traffic.
- Find root cause Compare request traces across the deploy boundary and reconcile them with pool metrics.
- Prevent recurrence Gate future releases on saturation and timeout signals with agreed halt rules.
Actions taken
- Reverted the connection pool configuration in place, leaving the schema migration untouched, and confirmed recovery against live bid traffic before closing the bridge call.
- Posted status to carriers and the account team every twenty minutes with plain language about lane impact, and offered a manual re-bid path for windows that had already closed.
- Wrote a postmortem within three days that separated timeline facts from hypotheses, and assigned each open question an owner and a review date.
- Added a canary stage that compares connection pool saturation and carrier-facing timeout rates against baseline, with automatic halt rules agreed with operations.
Results
Service returned to normal within the hour, and bid success rates recovered to their usual level before the settlement run began. Carriers who had lost windows received manual re-bid access, and the exchange recorded no lane allocations that finance had to reverse. The postmortem identified two process gaps: the release had no pool-level load test, and the rollback runbook assumed schema and configuration would move together. The canary stage has since halted two later releases on saturation signals, both caught before carriers saw errors. Operations now asks for pool metrics in every release note, which has shortened change review.
Questions for discussion
- Which evidence would have been enough for the team to roll back the schema as well, and what would that have cost in the settlement window?
- How should a team decide when a partial fix is acceptable while the root cause is still uncertain?
- What belongs in a release gate for a shared service, and who should own the thresholds?
Case Study 2
Reconciling recall rates across scheduling and billing feeds
Harrow Point Dental Group, a practice network of fourteen clinics that shares scheduling and billing systems.
The situation
Harrow Point's operations group prepares a monthly report on patient recall, the share of patients who return for a scheduled follow-up within the target interval. Two reports for the same period disagreed by about four points. The scheduling system put recall near seventy-one percent, while the billing warehouse put it near sixty-seven. Both reports were built by the same analytics team, both were circulated to the clinical directors, and both carried a note that the figures were final. Each clinic manager had already chosen the number that fit their narrative, which made the disagreement a political problem as well as a technical one.
The challenge
The business needed one recall figure before the quarterly staffing plan was set, and the directors had asked for a single version of the truth in writing. The billing extract had been rebuilt twice in the past quarter, and nobody could say with confidence which rebuild the warehouse had loaded. Some appointment types were mapped to billing codes that did not exist in the scheduling system, and a clinic that had migrated its system recently appeared in one source but not the other.
Analysis
The directors wanted a number they could staff against, not a perfect audit, so the first task was to define recall in a way both systems could honor. The definition in the report template was written for the scheduling system, and the billing side had been approximating it from visit codes. Lineage was the strongest evidence available. The team traced a sample of two hundred patient records from each report back to their source rows and found that the billing warehouse dropped follow-ups booked under a legacy appointment type. Counts in the scheduling system matched the clinic calendars at unmigrated sites, which gave the team a reliable anchor.
Time and access shaped the work. The warehouse team could not change the production loader until the next maintenance slot, and the clinics could not wait for a re-platform. The team therefore built a reconciliation table that compares row counts and recall rates by site and appointment type every night, then flags any site whose gap exceeds a tolerance agreed with the directors. Stakeholder impact was uneven: clinical leads cared about the figure for staffing, while the billing group worried that a new monitor would expose errors in their revenue reporting. Early conversations were tense, but the tolerance gave both groups a shared threshold independent of who was right.
Actions taken
- Pulled a sample of source rows for each disputed site and traced them through every transformation, recording where counts diverged and which rule caused the change.
- Agreed on a written recall definition with the clinical directors, covering the eligible visit types, the target interval, and how legacy appointment codes map to current ones.
- Added nightly reconciliation checks that compare row counts and recall rates by site, and that block publication when a gap exceeds the agreed tolerance.
- Asked the billing group to fix the legacy code mapping in the next maintenance slot, and kept the old figure marked as provisional until the fix was verified.
Results
The directors approved a recall figure of sixty-nine percent for the staffing plan, with a footnote on the legacy code gap. The nightly checks caught a second divergence in the first month, at a clinic whose calendar had been re-keyed after a system change, and the fix took one day rather than a week. Report conflicts in the monthly pack dropped to zero for two cycles. The billing group now reviews reconciliation flags before close, and the clinical leads have stopped keeping private versions of the figure. The definition document is now shorter than the original draft.
Questions for discussion
- When a reconciliation tolerance is agreed, who should be accountable for explaining a gap that falls just inside it?
- How much lineage detail should a report owner see before accepting a figure, and what should they do with the figure while it is still provisional?
- Which stakeholder would be most affected if the monitoring exposed errors in a revenue report, and how should their concerns shape the rollout?
Case Study 3
Scoping a suspected service account compromise
Quillfeather Credit Union, a member-owned lender with four branches and a small in-house IT group.
The situation
Quillfeather's infrastructure team runs a backup administration account that can read member loan files and modify retention policies. The account is meant for scheduled jobs only, and it is restricted to two servers. During a routine review of authentication logs, an analyst noticed interactive logons from that account, outside its usual maintenance window, originating from a hosting provider's address range. Two of the logons had succeeded, and a third had been followed by a large export from the file share. The analyst flagged it to the on-call security lead, who had been reviewing alerts for most of the day.
The challenge
The team had to decide quickly how much to disrupt. Disabling the account would stop the activity but could break overnight backups that the credit union depends on for member recovery, and a live export might still be running. Forensic images of the two servers would take most of a shift, and the regulator's reporting rules meant that any member data exposure would need a documented timeline. Nobody yet knew whether the credentials had been reused elsewhere.
Analysis
The priorities were ordered before the facts were complete. Member trust and regulatory obligations weighed more heavily than the cost of a few hours of backup delay, but the team could not yet prove scope. The evidence came from three sources: authentication logs, the file share audit trail, and the hosting provider's address records, which said little about intent. Logs on the two servers were the most reliable, but they rotate every few days, so the team exported them before doing anything else. The export was hashed and copied to write-once storage, which made it usable later if counsel asked for it.
Containment was a sequence, not a switch. The team first restricted the account's network path to the two servers, which kept backups running at unaffected sites, then rotated the password and revoked active sessions. Full disablement waited until the backup owner confirmed a manual job could cover the overnight window. Stakeholder impact fell unevenly. Member services wanted affected loans identified within the day, the retention team needed to preserve policy changes made under the account, and the board needed a single account of decisions. The security lead wrote each step with its time and the name of the person who approved it, so the record stood on its own.
Actions taken
- Exported authentication logs, the file share audit trail, and servers' process lists, then hashed each file and stored copies in write-once storage with a custody note.
- Restricted the account to a single bastion path and revoked active sessions, then rotated its credentials through the vault rather than by email or chat.
- Scoped the exposure by searching the file share for the account's reads over the prior thirty days, and listed every member record touched for legal review.
- Notified the security officer and counsel with a written timeline, and recorded which steps were taken under standing authority and which awaited their approval.
Results
Scoping found that the account had read forty-one loan files before the export, and that the export itself had been stopped by the restriction in its second minute. No evidence showed credential reuse on other systems, though the question stayed open. Member notices were limited to the affected records, and the regulator received a timeline that matched the preserved logs. The account was retired and replaced with two scoped service identities, one per job. The authorized response procedure now names who may disable an account, isolate a host, and sign off each step.
Questions for discussion
- What would the team have lost by disabling the account in the first hour, and could a narrower control have protected the same members?
- How should a response team record decisions made under standing authority so that the record holds up when counsel reviews it?
- Which parts of this containment sequence depended on people being available, and how would the procedure change if they were not?
Case Study 4
Choosing a sales path under peak demand
Mosswood Ticketing, a live-events ticketing service that sells seats for about 300 venues.
The situation
Mosswood sells tickets for concerts, theater runs, and sports events through a web and mobile storefront. Its current platform runs in one cloud region behind a single load balancer, with a relational database that holds seat inventory and order state. Sales for high-demand events arrive in bursts within the first few minutes after an on-sale time. During the last two major on-sales, checkout latency at the ninety-fifth percentile reached about nine seconds, and abandonment climbed sharply. Finance estimated that each minute of degraded checkout cost roughly 2,400 dollars in lost margin on a typical large event. Leadership wanted the cost of peak handling to stay flat while latency fell.
The challenge
The three options on the table had very different risk profiles. Scaling the existing database vertically was quick but had a ceiling that the team had already approached. A queue in front of checkout would protect the database but changed the purchase experience, and the venue partners had contracts that specified how seats were held. A second active region would solve latency for distant buyers but doubled the surface for seat inventory conflicts.
Analysis
The business goal was narrower than general speed. Mosswood needed checkout to hold steady for the first ten minutes of the largest on-sales, because that window carried most of the revenue. Evidence came from load tests and from production traces of the last two events. Replays showed that the database's connection limit, not CPU, was the first constraint hit, which made vertical scaling less useful than it appeared. Traces also showed that roughly two thirds of checkout time in the peak went to seat hold writes that could tolerate eventual confirmation for the browsing layer, though not for final payment. That distinction made it possible to split the problem.
Operations ruled out a full active-active build before the next major on-sale. The on-call rota had two engineers who had run database failover, and a second region needed a payment processor review that took weeks. Security added its own limits: seat inventory and payment data would need the same controls everywhere, and the partner contracts required that seat holds be auditable. Partners cared mostly about fair access and sold-out accuracy, so the team needed any queue to preserve first-come ordering. The chosen option adds a queue for seat holds only, a read cache for event pages, and a documented path to a second region once failover drills are routine.
- 1Single region baselineOne database and one load balancer, with latency spikes at each on-sale start.
- 2Bounded seat hold queueCaps writes to the database and preserves first-come ordering for popular events.
- 3Read cache for pagesServes event pages and counts from memory, with payment always reading the primary.
- 4Second active regionLowest latency for distant buyers, but doubles inventory conflict and audit scope.
- 5Quarterly failover drillKeeps the region option credible and exposes runbook gaps before a real outage.
Actions taken
- Moved seat hold writes behind a bounded queue with first-come ordering, and set a visible wait estimate so buyers did not refresh and add load.
- Added a read cache for event pages and availability counts, with a short expiry and a rule that payment and seat confirmation always read from the primary.
- Raised the primary database's connection limit only after a load test confirmed the pool settings, and kept a rollback plan that did not require a schema change.
- Scheduled a failover drill each quarter and wrote the second-region build as a gated milestone, with the compliance review started before any capacity is bought.
Results
The queue and cache went live ahead of the next major on-sale. Checkout latency at the ninety-fifth percentile dropped to under two seconds in the peak window, and abandonment returned to its earlier level. The team did not buy a second region, and the monthly infrastructure bill stayed near its flat-cost target. Two problems surfaced after launch. A cache expiry set too long showed sold-out events as available, caught by a partner and fixed within a day. The failover drill exposed a runbook gap that the team has since closed. Second-region work continues on its own schedule.
Questions for discussion
- At what level of cost growth would the team be justified in moving to a second region, and who should decide that threshold?
- How should a cache policy for availability counts be tested before it reaches buyers?
- What evidence would convince partners that a queue preserves the fairness they were promised?
Case Study 5
Piloting an automated maintenance triage model safely
Larkspur Property Management, which handles maintenance for about 2,300 residential units across three cities.
The situation
Larkspur's maintenance desk receives requests by phone, text, and portal form, and dispatchers sort them by hand each morning. A vendor offered a model that reads each request, assigns a priority and trade, and proposes a technician slot. The vendor's pilot summary claimed a large drop in time to first response, but the figures came from a single property with newer buildings and a dispatcher who had tuned the categories. Larkspur's operations director, Renata, wanted faster response for residents with urgent problems, such as no heat or water leaks, and she did not want the model to decide who waited longer. Dispatchers were already stretched during the cold months.
The challenge
The question was not whether the model could be fast. It was whether its errors would fall on the same buildings and residents every time. Dmitri, the dispatch lead, had seen earlier automation route calls from older buildings to the least experienced crews. Any rollout had to be reversible in a day, and it had to work with the dispatch software the team already used, which had no native hook for model suggestions.
Analysis
The evidence was weaker than the summary suggested. The pilot covered one property, one season, and a dispatcher who had reworded categories to match the model's outputs. That made the reported gain hard to transfer. Larkspur therefore asked for a different kind of evidence: a retrospective run on six months of past requests, scored against what technicians actually found on arrival, with results broken out by building age, neighborhood, and request channel. The business objective was to shorten response for urgent requests without lengthening it for anyone else, so the metrics had to measure both sides. Accuracy alone would hide an uneven distribution of errors.
Human oversight had to be designed into the workflow, not added as a policy. The team chose a shadow period in which the model proposed priorities and slots, dispatchers recorded their own choice first, and the difference was logged without affecting any dispatch. Once live, any suggestion with an urgent category required a dispatcher confirmation, and residents with heat or water emergencies were never routed on model output alone. Technicians and residents bore the consequences of errors, so their complaints became part of the monitoring, not an afterthought. The vendor agreed to expose the reasons behind each priority so that dispatchers could challenge them.
- First weekRetrospective comparison runs on six months of requests, no live routing yet.
- Weeks two through fourShadow mode records dispatcher choices beside model suggestions without affecting dispatch.
- Month twoStandard requests route through the model at two properties, with fallback ready.
- Month threeThresholds reviewed; expansion decision made only if error gaps stay within limits.
- Each quarter afterFairness review by building group, with the vendor explaining any drift in priorities.
Actions taken
- Ran the retrospective comparison on six months of requests, reporting error rates by building age, neighborhood, and channel before any live routing was discussed.
- Set written thresholds before the pilot: urgent-request misses below a fixed ceiling, no group of buildings with error rates far above the median, and dispatcher override rates tracked weekly.
- Limited live routing to standard requests in two properties, with an automatic fallback to manual sorting if override rates or complaints crossed the agreed lines.
- Gave dispatchers a visible reason field for every suggestion, and reviewed a random sample of accepted and rejected suggestions with the dispatch lead each fortnight.
Results
The retrospective run showed that the model was faster on urgent requests in most buildings, but it sent older buildings to a narrower set of crews and misclassified water leaks in one neighborhood about one time in six. Larkspur did not roll out the model across the portfolio. The two-property pilot went live after the thresholds were revised, and response time for urgent requests fell modestly while dispatcher overrides settled at a rate the director accepted. The neighborhood problem traced to a vocabulary gap in resident messages, which the vendor corrected before expansion.
Questions for discussion
- Which error pattern would you want a retrospective test to surface before any live routing, and why is it hard to see in a single pilot?
- How should a dispatcher's override be weighed when it may reflect either a model error or local knowledge the model lacks?
- What would justify expanding a partly validated automation tool to the full portfolio, and who should sign that decision?
Learning Resources
Technical CASE STUDIES
China, India, United States, Indonesia, Pakistan, Nigeria, Brazil, Bangladesh, Russia, Japan, Mexico, Philippines, Vietnam, Egypt, Turkey, Germany, United Kingdom, France, Italy, Canada, South Korea, Australia, Spain.