Software Quality Assurance and Test Automation Case Studies
Technical5 Case StudiesUpdated Sep 2026
The five cases below come from teams whose work sits between building software and proving that it works. Each is set inside a fictional organization and follows a decision made with incomplete evidence, often under deadline. Every case returns to the same four lenses: system requirements, performance evidence, access controls, and maintainability. Those lenses matter because quality problems rarely announce which part failed. Teams that can name the requirement, the measurement, the permission, and the maintenance burden usually recover faster than teams that start by arguing about blame.
Case Study 1
Dispatch slowdown after a Thursday release
Harrowgate Freight Systems, a logistics software vendor whose dispatch platform coordinates pickups for regional trucking carriers.
The situation
Harrowgate released a change to its dispatch platform on a Thursday afternoon. The change reworked how route assignments were cached so that carriers could reassign loads without a full page refresh. By early evening the assignment endpoint was responding in several seconds where it had previously taken under two hundred milliseconds, and a holiday weekend volume increase was arriving on top of the slowdown. Dispatchers began reporting that drivers were receiving the same job twice. The release had cleared the automated regression suite, which checked functional behavior at low concurrency and never exercised the cache during the evening batch window.
The challenge
The on-call engineer, Priya, had to restore service for carriers who were already scheduling trucks for the weekend. Rolling back the release was the obvious move, but the deployment had also run a schema change that two later jobs depended on, and nobody on the rotation was certain a rollback would leave those rows readable. Production logs were limited to a small access group, and the two engineers who understood the caching layer were both off shift. Leadership wanted a root cause before anyone touched the deployment.
Analysis
Teams grounded in Software Quality Assurance and Test Automation tend to treat a release as a set of stated requirements, including the non-functional ones. Here the requirement for the assignment endpoint described only behavior, that a carrier can reassign a load, with no response time target and no concurrency level. That gap let the regression suite pass while the feature failed under evening load. Priya's first useful step was to pull response time percentiles for two hours either side of the deploy, not averages, because the average hid a tail of timing out requests. The break fell exactly at the deployment timestamp, which pointed the team at the change rather than at traffic.
The access question was about who could read what, and how quickly. Production logs were restricted to a small group, which turned the first hour into a queue. Read access to aggregated latency and cache metrics for the rotation, without raw log access, would have let the first responder separate a cache problem from a database problem without waiting. Maintainability showed up in the runbook, which described the caching layer from a design document written before the last two refactors. Nobody had updated the eviction section, so the team's steps were partly guesswork until the author was reached.
- 1Check the release timelineMatched the slowdown start to the deployment timestamp, not to traffic.
- 2Restore with a flagDisabled the new cache path while leaving the schema change in place.
- 3Confirm the causeTraced stale entries to a cache key missing the assignment version.
- 4Gate future changesLoad testing at evening peak now runs before caching changes reach production.
Actions taken
- Add a release gate that runs a load test at evening peak concurrency for any change touching caching or assignment logic. The gate reads its threshold from the written requirement.
- Grant the on-call rotation read access to aggregated metrics dashboards that separate cache, database, and network latency, and keep raw log access behind a short approval step with an expiry.
- Rewrite the caching runbook with the eviction and rollback steps, assign each section an owner, and review it at the close of every release cycle so the document reflects the code that actually shipped.
- Before the next change to the schema, confirm in staging that the previous version can be restored with the new rows still readable, and record that check in the release checklist.
Results
Priya restored service by disabling the new cache path with a configuration flag rather than rolling back, which kept the schema change in place. Assignment latency returned to its previous range within the first hour, and the duplicate job reports stopped once the stale entries expired. The root cause review traced the problem to a cache key that did not include the carrier's assignment version. The release gate caught a similar defect during the following month in a staging run, before any carrier saw it. Support tickets for the weekend numbered about forty, nearly all about jobs already corrected.
Questions for discussion
- Which of the four actions would have caught this incident earliest, and what failure would it still have missed?
- How should an on-call rotation weigh read access to raw logs against the time it costs to wait for approval during an incident?
- When a runbook section is known to be out of date, what is the least a responder should verify before acting on it?
Case Study 2
Conflicting no-show rates across clinic reports
Brightwater Clinics, a regional network of eleven outpatient clinics whose reporting group consolidates appointment and billing records each night.
The situation
Brightwater's operations group began receiving two versions of the monthly no-show rate in the same week. The scheduling dashboard reported a rate about four points higher than the figure in the finance pack, and clinic managers were using both to justify different staffing decisions. Both came from one appointment table but used different filters. The nightly consolidation job pulled appointments from four scheduling systems that had been merged, and a handful of clinics still recorded cancellations under a legacy status code. Roughly 3,400 appointment records per month were affected by the mismatch, enough to move the headline figure.
The challenge
Krisztina, the data lead, had to decide which figure to correct first without freezing reporting for the clinics. Finance had already sent the pack to the regional board, and changing the definition would make the trend line look like a break in the series. She also could not tell from the schema whether the legacy status code had been mapped consistently across all eleven sites. The transformation logic lived in three places, and only one of them had an owner who was reachable that week.
Analysis
A reporting definition is a requirement, even when nobody wrote it down as one. Here the no-show rate existed in two forms: one counted cancellations under the legacy code as no-shows, and the other excluded them. Neither was documented as the agreed definition. Krisztina treated the question as a specification problem first and a data problem second, and she asked clinic leads to state what each staffing decision actually depended on. Her evidence came from row counts reconciled between source and target at each stage, plus a sample of two hundred records checked by hand. The reconciliation showed where the gap opened, which mattered more than any single total.
Access was the quiet constraint. The transformation code was editable by anyone with access to the warehouse project, which meant a well-meaning analyst had altered a mapping table during the investigation with no record of the change. Krisztina moved the mapping into a version-controlled file that only the data platform group could merge, and gave analysts read access to the mapping and its history. Maintainability improved for a similar reason. With the three transformation locations collapsed into one documented module, the next change to a status code could be reviewed in a single pass rather than traced through three owners.
- Start of the weekTwo no-show figures reached managers from different sources.
- Second dayReconciliation by site and status code located the gap.
- Third dayMapping file moved under version control with a named owner.
- Third weekCorrected definition published with an annotated finance series.
- End of the quarterWeekly comparison caught a new divergence, fixed before the monthly pack.
Actions taken
- Reconcile appointment totals against the source system by site and status code, and fail the job when the gap exceeds an agreed tolerance, so mismatches surface before anyone builds a report on them.
- Write the no-show definition as a single signed-off statement that names every status code it includes and excludes, and attach it to each dashboard so readers see the rule beside the number.
- Move the status mapping into a version-controlled module with a named owner and a required review for any change, and give analysts read access to its history.
- Add a weekly comparison between the finance pack and the scheduling dashboard for the next quarter, with a short written note whenever the two diverge by more than an agreed margin.
Results
The reconciliation showed that the legacy status code accounted for most of the gap, and two clinics had entered cancellations as reschedules. The corrected definition went out in the third week, with the finance series annotated rather than restated, so the board could see when the change took effect. Four clinics revisited hiring requests that had relied on the earlier figure. The weekly comparison later caught one more divergence, from a new scheduling integration, and it was fixed before the monthly pack was assembled. Clinic managers now quote a single figure with its definition attached.
Questions for discussion
- When two reports disagree and both were built correctly from the same source, what does the team need to ask first about the definition?
- Which parts of a reconciliation should be automated, and which still need a person who knows the clinics to review them?
- How should a data owner balance a correction against the need to keep a published series stable for the people relying on it?
Case Study 3
Unexpected export jobs from a service account
Quillfield Payroll, a payroll processing company that runs hosted pay calculations and direct deposit files for small employers.
The situation
Quillfield's security monitoring flagged a service account that had begun exporting employee bank detail files outside its normal schedule. The account belonged to the payment file generator, a component that should only write files to an internal staging bucket and never read from the customer data store. Over two nights the account made several hundred read requests against a table it had never touched before, from an address range assigned to a cloud region the company did not use. The monitoring tool had been configured to alert on volume rather than on new destinations, which is why the alert arrived late. The rotation had not confirmed whether the credentials were misused or simply reused by a deployment script.
The challenge
Ayodele, the security lead, had to decide whether to disable the account now, which would stop any real exfiltration but could also halt payroll runs scheduled for the following morning. Preserving evidence meant not rotating the credentials before the logs were copied, and the copying itself had to happen without the account owner's team changing the retention settings. She also had only partial visibility into which deployment scripts used the account, and the person who maintained them was on leave.
Analysis
The requirement that should have caught this was written in the design document in plain terms: the file generator reads nothing from the customer store. No test or policy enforced it, so the account's permissions had grown to include read access during an earlier integration and nobody removed it. Hourly request counts over the previous thirty days showed no spikes, so a volume alert had nothing to fire on. A baseline of destinations and table names would have been more sensitive. Ayodele asked for a comparison of the account's observed access against its documented requirements, and the gap between the two was the first finding.
Access controls explained most of the exposure. The service account had been granted a broad role during an integration that was later retired, and the role was never narrowed because the retirement ticket covered the integration code and not its permissions. Maintainability was the second weakness. Three deployment scripts shared one credential file, so rotating the secret meant coordinating changes across repositories that different teams owned. A stronger arrangement would give each job its own scoped identity with an expiry, plus a quarterly review that compares granted permissions against the design document. None of that is exotic, but it needs a named owner who acts on the review.
Actions taken
- Before any credential change, export the account's authentication and access logs for the prior thirty days to write-once storage, with the hash recorded in the incident ticket.
- Disable the account's read permission to the customer store immediately, since that permission was never part of the documented requirement, and leave its staging write access in place until the payroll run is confirmed.
- Give each deployment job its own scoped identity with an expiry, and retire the shared credential file once the replacements have run cleanly through one full payroll cycle.
- Write a short authorized-response note that lists who may disable accounts, who may read the logs, and which steps need a second approver, and keep it with the on-call runbook.
Results
The logs showed the account had read 2,300 customer bank detail rows over two nights, and the exported files sat in the staging bucket with no external transfer recorded. Ayodele disabled the read permission after the copy completed, and the payroll run went out on schedule under a fresh scoped identity. The deployment script that had reused the shared credential was reading the table during a test nobody had removed. The team finished the review in the second week, retired the shared file, and added destination-based alerting, which fired once later on a legitimate and announced change.
Questions for discussion
- What evidence would have been enough to justify disabling the account before the payroll run, and who should have been empowered to make that call?
- How should a team decide which logs to preserve first when retention settings are short and an investigation is still open?
- Which of the authorized response steps in this case depended on one person being available, and how would you remove that dependency?
Case Study 4
Caching and queueing options for on-sale spikes
Greyhaven Tickets, a ticketing platform for mid-sized concert venues and theaters that sells access passes through its own web and mobile clients.
The situation
Greyhaven's ticketing platform handles routine sales for a few hundred venues, but a popular on-sale can send more purchase attempts in a few minutes than a normal week produces. The checkout service had been built as a single application with one relational database, and the team had added read replicas and a cache for seat maps. During the last large on-sale, seat map requests stayed fast but purchase confirmations slowed to several seconds, and about 1,200 customers saw a payment error after the charge had already posted. Leadership asked for a design that would hold up during the next major on-sale without raising the platform's monthly cost.
The challenge
Jamal, the platform architect, had three options on the table and a deadline set by the venue contracts. A message queue in front of checkout would smooth bursts but add a new operational system that the small platform team had never run. Moving sessions to a managed key-value store would cut database load but would need a data migration during a period with no slack. Scaling the existing database vertically was cheap to start but had a known ceiling. The team needed to pick one before the contract dates.
Analysis
The useful move was to turn the vague goal of holding up during on-sales into measurable requirements. Greyhaven wrote down the peak purchase attempts per minute from the last two events, a target confirmation latency at that rate, and an acceptable error rate. With those numbers in hand, the team could load test each candidate against the same workload. The load tests showed that the vertical database option met latency at the old peak but degraded once traffic reached about double it. The queued design held latency steady at double load, but customers waiting through the queue drain saw tails long enough that support expected calls.
Access and operability decided between the two finalists. A queue adds a broker with its own credentials, network paths, and encryption settings, and each of those needs an owner and an audit trail. Payment data has to stay within the cardholder environment, so the team would need to confirm which components could see message contents. The managed key-value option reduced the database's exposure but moved session data into a service that the security group had not yet reviewed. Maintainability favored the design the team could explain on one whiteboard, which was the vertically scaled database with a read replica and a waiting-room page, though it had the least headroom.
- Measure peak demand Record purchase attempts per minute from past on-sales and set targets.
- Load test options Run each candidate against the same recorded workload at double peak.
- Review security surface Count new credentials, data paths, and components that touch payment records.
- Compare after events Compare live latency with the test results and adjust the next capacity plan.
Actions taken
- Write the workload targets as numbers before any further design work, covering peak purchase attempts per minute, confirmation latency at that rate, and the acceptable error rate.
- Upgrade the database instance and add a read replica for seat maps, then run the full sales workload against staging before the contract dates are confirmed.
- Build a waiting-room page that admits buyers in measured batches, and test it with simulated bursts that reach double the recorded peak.
- Record the queue design as a deferred option with its owner, its security review, and the operating skills it requires, so the next planning cycle starts from a concrete proposal.
Results
The team chose the vertically scaled database with a read replica for seat maps, a waiting-room page for on-sale traffic, and a short-term limit on purchase attempts per account. The database upgrade and a full waiting-room test finished in the third week. The next major on-sale brought about 1,800 purchase attempts per minute at peak, above the earlier peak, and confirmations stayed within the agreed target. The queue was deferred to a later planning cycle, when the platform team had someone who could operate a broker without support. Monthly hosting costs rose by a modest amount, inside the approved envelope.
Questions for discussion
- How would the recommendation change if the venue contracts had allowed a further two months, and what would the team have needed to verify before committing to the queue?
- Which of the workload targets in this case were measured and which were estimated, and how should a team treat that difference in a capacity review?
- What security review would you require before adding any new component that touches payment records, and who should own the outcome?
Case Study 5
Evaluating an automated test generator before rollout
Pelletier Insurance Services, a claims administration firm that processes property and auto claims for several regional carriers.
The situation
Pelletier's claims platform had a regression suite of 4,700 automated tests, and the team spent most of each sprint writing and repairing them by hand. A vendor offered a tool that records a session through the claims interface and proposes test cases, including the expected values for each screen field. Its materials promised a cut in authoring time, and procurement wanted a decision before the next budget cycle. The tool generated its assertions by reading current application output, which meant any defect present during recording would be written into the expected results. The team had no information about how the vendor stored recorded claim data, which included claimant addresses and medical summaries for some lines of business.
The challenge
Tomasz, the QA lead, had to recommend whether to run a pilot at all. A pilot on live recordings would expose real claimant data to a tool whose storage terms were unclear, but a pilot on synthetic claims would not show how the tool handled the messy screens that caused most of the team's repair work. Two senior testers were skeptical, and one was openly worried that a generated suite would look complete while missing exactly the cases that mattered.
Analysis
The first requirement was not the tool's speed but what a correct test means for this product. Pelletier defined a small set of claim paths that had to be covered, such as partial payments, subrogation holds, and reopened files, and each path had an expected outcome written by a claims specialist rather than copied from the application. The team measured the share of required paths covered by generated tests, the rate at which generated assertions matched specialist expectations on a sample, and the number of false passes, meaning tests that stayed green while the expected outcome was wrong. Authoring time was recorded too, but as the last column, not the first.
The vendor's materials said least about access and upkeep. Recorded sessions contained personal data, so the pilot needed a written statement of where recordings were stored, how long they were kept, and which vendor staff could open them. Pelletier's security group required that recordings be masked at capture for any field tagged as personal, and that masking be tested as its own requirement. On maintainability, generated suites tend to encode the screen layout at the time of recording. The team asked how many tests would break after a routine interface change, and whether a regenerated test would overwrite manual edits. Those answers mattered more than first-month authoring numbers.
Actions taken
- Write the acceptance criteria before the pilot starts, including the required path list, the sample size for assertion review, and the false pass threshold that stops rollout.
- Require a named specialist to approve every generated expectation for personal-data and payment paths, and record the approval in the test's metadata.
- Keep recordings inside Pelletier's environment with masking applied at capture, and test the masking as a separate requirement before any recording leaves the staging network.
- Roll out in stages, starting with low-consequence screens, and pause expansion if the false pass count rises or if a routine interface change breaks more than an agreed share of generated tests.
Results
The pilot ran for the length of two sprints on a set of recorded sessions covering the required claim paths. Generated tests covered most of the required paths, but specialists rejected a meaningful share of the generated expectations, mostly on partial payment rounding and reopened file dates. Two generated suites would have passed while asserting the wrong outcome, which the false pass measure caught before any rollout decision. Pelletier adopted the tool only for low-consequence screens, with specialist approval required for everything else. Authoring time fell for those screens, though by less than the vendor's materials had suggested.
Questions for discussion
- What evidence would convince a skeptical senior tester that a generated suite is missing cases, rather than simply reflecting the application as it stands?
- How should a team set the false pass threshold when the expected outcomes themselves come from a small number of specialists?
- Which would you measure first in a pilot of an automation tool, coverage, assertion accuracy, or maintenance cost, and why?
Learning Resources
Technical CASE STUDIES
China, India, United States, Indonesia, Pakistan, Nigeria, Brazil, Bangladesh, Russia, Japan, Mexico, Philippines, Vietnam, Egypt, Turkey, Germany, United Kingdom, France, Italy, Canada, South Korea, Australia, Spain.