Web Application Security Case Studies
Technical5 Case StudiesUpdated Sep 2026
The five cases below come from fictional organizations, but the pressures in them will be familiar to anyone who has shipped a web application. Each one starts with a concrete failure or a hard decision: a slow release, conflicting numbers, a possible compromise, a scaling squeeze, or an automation proposal with unclear risk. The analysis in each case follows the same four threads: system requirements, performance evidence, access controls, and maintainability. Read them in order if you like, but the decisions matter more than the sequence.
Case Study 1
Search slowdown and an exposed route after release
Larkspur Scheduling, a fictional online appointment platform used by eleven independent physical therapy clinics.
The situation
Larkspur runs a booking portal where patients reserve appointments and clinic staff manage schedules. It is a single Python service behind a load balancer, backed by Postgres and a Redis cache for sessions. Releases ship every other week after unit tests and a short staging check, but nobody runs load tests. In the second week of a cycle, a change to availability search added a join against full appointment history. Staging held about 4,000 appointments, so nothing looked wrong. Production held more than 2,000,000, and within the first hour p95 search latency rose from under 300 milliseconds to several seconds.
The challenge
The on-call engineer, Marisol, had two problems that pulled in different directions. Restoring service meant rolling back quickly, but the release also carried a schema migration that had already run. Meanwhile the access logs showed an unauthenticated route, left behind from a debugging session, that returned full appointment records when given a numeric id. Someone had found it during the slowdown and was hitting it in bulk. Rollback alone would not close the exposure, and blocking the route alone would leave the slow query running.
Analysis
The first useful step was separating what the system was required to do from what it happened to do. Larkspur's requirements said search must return within 500 milliseconds at the ninety-fifth percentile, but nobody had tested that at production volume, so the requirement existed only on paper. Marisol pulled the query plan from the database and compared row estimates with actual counts, which showed the join scanning the whole history for every keystroke in the search box. That is performance evidence in its plainest form. The access logs were the second source, and they told a different story about the same route.
Teams grounded in Web Application Security tend to treat access control as part of the same incident, not a separate ticket. The unauthenticated route had been added for a debugging session and never removed, which is a maintainability problem as much as a security one. Nobody on the current team knew it existed until the logs showed it. Reviewing the route table against the framework's decorators would have caught it in the release review. The lasting fix was a build test that fails whenever an API route lacks an explicit authorization rule, so the next engineer cannot rely on memory.
- Minute zeroRelease deploys with an unindexed join against full history.
- First hourp95 search latency climbs past two seconds and alerts fire.
- Second hourAccess logs reveal bulk requests against an unauthenticated route.
- Third hourRoute blocked at the load balancer and API keys rotated.
- Next morningForward migration ships, restoring the index and fixing the query.
Actions taken
- Marisol blocked the unauthenticated route at the load balancer within twenty minutes, rotated the keys used against it, and confirmed in the logs that the bulk requests had stopped.
- The team skipped a full rollback. They wrote a forward migration that restored the previous index and reverted only the search query, keeping the schema change the rest of the release needed.
- Before the next release, the team added a load test that replays a recorded week of production searches against staging, with a p95 threshold that fails the pipeline when a change crosses 500 milliseconds.
- Each release checklist now requires a named owner and an authorization rule for every new route, plus a removal date for anything added only for debugging.
Results
Search latency returned to its normal range by the end of the day. The logs showed no confirmed patient record exports, though the team notified the affected clinics as a precaution and reviewed access going back to the route's creation. The forward migration kept the release intact, so the remaining features shipped on schedule the following week. The load test caught a second, smaller regression in the next cycle, which was fixed before it reached production. Two clinics asked for a written account of the exposure, and the team provided one with the timeline attached.
Questions for discussion
- Which performance requirements in your own system exist only as a document, and what production data would you need to test them?
- If the fastest rollback would have left an exposed route running, how would your team decide which action to take first?
- What would it take for your release checklist to name an owner for every route, and who would be the right person to keep it honest?
Case Study 2
Conflicting failed-login reports and the pipeline behind them
Tallow Creek Marketplace, a fictional online marketplace for handmade goods, with a small platform team and a separate security group.
The situation
Tallow Creek runs a marketplace where about 38,000 sellers manage storefronts through a web application with its own sign-in, and an identity provider that handles multi-factor checks. Two groups report on sign-in activity. The security team reads raw access logs from the web tier, while the product group reads events emitted by the identity provider. Each Monday both groups present numbers for failed logins, and the two figures had drifted apart for several weeks. The gap was large enough that the leadership group had asked whether to tighten lockout rules, and nobody wanted to act on a number that might be wrong.
The challenge
The obvious explanation was duplication, but the counts did not behave like duplicates. The web tier log showed about 4,200 failures in a typical week, while the identity events showed 1,900, and the ratio shifted on weekends. Someone had also noticed that a regional migration in the second month had changed the timestamps from UTC to local time in one feed but not the other. Before anyone could decide which number to trust, the team needed to know which records were being counted, and why.
Analysis
Lineage came first. The team drew the path each number took, from the raw web log or identity event through a shipping agent, a parser, a deduplication step, and a weekly aggregate. The diagram exposed something the dashboards hid: the parser read the client address from a forwarded header, which the content delivery network set for every request, so failures from one large office network were being collapsed into a single address. Requirements for the report had never said whether the figure was attempts or unique accounts, so each team had quietly chosen a different answer.
Validation should have happened at each transformation, not at the end. The team wrote reconciliation checks that compared record counts at each stage and flagged any day where the parser dropped more than a small share of input. Permissions were a separate problem. The security group had read access to raw logs but not to the parser configuration, so changes to the header handling had been made by the platform team without review by anyone who consumed the output. Monitoring that alerts on drift is only useful if the people who change the pipeline are on the same list as the people who depend on it.
Actions taken
- Pin the definition first: the team agreed that the headline figure counts unique sign-in attempts per account per hour, and wrote that rule into the report's header and the data dictionary.
- Rebuild the forwarded-header handling so the parser takes the client address only from the trusted proxy hop, and add a test with spoofed headers that must be rejected.
- Add reconciliation checks at each stage that compare counts from the raw source, the deduplicated set, and the final aggregate, and page the on-call analyst when the gap exceeds an agreed threshold.
- Give the security and product groups shared ownership of the pipeline configuration, with read access for both and a review requirement for any change to parsing or time handling.
Results
The reconciliation check caught the timestamp mismatch on its first run, and the team normalized both feeds to UTC before the next Monday report. After the header fix, the web tier count fell to roughly 2,100 failed attempts per week, close to the identity figure once the deduplication rule was applied consistently. Leadership deferred the lockout change until the numbers held for three straight reports. The security group now receives the same aggregate as product, with a note on which figures are raw and which are deduplicated.
Questions for discussion
- Which figures in your own reporting would change if you had to write down the definition of a single event before anyone counted it?
- When two teams disagree about a number, who should own the final answer, and what evidence would you require before changing it?
- How would you decide which reconciliation checks deserve a page at night and which can wait for a morning review?
Case Study 3
Suspicious administrator logins at a bookkeeping service
Verdant Ledger Associates, a fictional bookkeeping service that hosts invoices and payroll exports for about 1,200 small firms.
The situation
Verdant Ledger hosts invoices, expense reports, and payroll exports for its clients through a web application with a separate administrative console. On a weekday morning, a support engineer named Tomas noticed an administrator account signing in from an address the firm had never used, followed by a burst of downloads from a single session that touched several tenants. The application logs were retained for thirty days, and the web tier shipped them to a central store. Nobody yet knew whether the credentials were stolen, whether a session token had leaked, or whether the activity was a misconfigured integration.
The challenge
The response lead, Imani, had to protect evidence and stop the activity at the same time, and the two goals pulled in different directions. Revoking every session would lock out clients in the middle of payroll week. Preserving the exact state of the admin console meant not restarting its containers yet. Her list of pre-approved actions was short and written for a different kind of incident, so most steps needed a sign-off she could not get before morning.
Analysis
Scoping started with what the logs could and could not answer. The retention window was thirty days, enough to see the admin login pattern but not the account's earlier history, so the limit went into the written findings. Performance mattered in an unexpected way: the log store was slow to query during business hours, and a full export of download events took most of an hour. Scoping by tenant and session cut the query to a few minutes. Requirements for the logging pipeline had been written for uptime and debugging, not for incident questions, and this was the gap that showed.
Access controls were the part most likely to fail quietly. The administrative console accepted a session token that remained valid for eleven days after a password reset, which is a rule nobody remembered writing. Keys for the signing service lived in a configuration file that three people could edit, and none of them had recently reviewed who held a copy. Containment therefore meant more than blocking one address. The team revoked tokens issued before the reset, moved the console behind a VPN, and rotated signing keys in an order that kept unaffected tenants online.
- Scope Define accounts, tenants, and time window from logs.
- Preserve Snapshot images, replicas, and indexes before restarting.
- Contain Revoke stale tokens, restrict console access, rotate keys.
- Document Record authorized steps so responders can repeat them.
Actions taken
- Snapshot the admin console's container image, the read replica, and the relevant log indexes before any restart, and record the hash of each snapshot in the incident file.
- Revoke tokens issued before the reset, block the suspicious address at the edge, and move the admin console behind the VPN so it is no longer reachable from the public internet.
- Export the affected tenant list and download events into a case workbook, and have a second engineer check each query against the written scope before any client is told that data was accessed.
- Rotate the signing keys in the agreed order, record which integrations needed new credentials, and file a short note with the authorized steps so the next responder can repeat them without guessing.
Results
The review found no export beyond that one session, and the administrator's password had been reused from an unrelated service, which explained the first login. Two tenants were notified because their invoice PDFs had been downloaded, and the firm documented the timeline for their own records. Clients whose integrations used the old signing key saw a brief interruption that was announced in advance. The response file became the template for the next incident, and the scope question that had taken hours now has a standing query.
Questions for discussion
- What would your team lose if it had to restart a compromised service before preserving its state, and how would you weigh that loss against the risk of continued access?
- Which of your administrative sessions would still be valid after a password reset, and how would you find out without waiting for an incident?
- How should a response plan name the people who may authorize client notification, so that the decision does not wait for a single manager to return?
Case Study 4
Choosing a lookup architecture under latency and cost pressure
Ferrule Freight Tracking, a fictional shipment status service with a public tracking page and a partner API, run by a six-person engineering team.
The situation
Ferrule serves about 14,000 shippers and carriers through a public tracking page and a partner API. Traffic rose steadily through the last several quarters, and p95 latency on tracking lookups crossed the target during peak weekday hours. The application is a Ruby monolith with one primary Postgres database, a read replica, and a Redis cache. The team of six can maintain the monolith, but hosting costs have outpaced revenue, and two large carriers have warned they will move integrations if lookups stay slow. Security requirements are strict because the API returns shipment contents tied to individual consignees, so any redesign must keep tenant isolation intact.
The challenge
The architecture group had to choose among three designs before the next budget cycle. Each promised lower latency, but they asked for different things from the team: caching public pages would not help authenticated lookups, splitting the read path would add deployment and access rules, and a service mesh would move complexity into every network call. The team needed a comparison that treated cost, tenant isolation, and the ability to recover from failure as first-class criteria, not afterthoughts.
Analysis
Requirements came before options. The team wrote down what the lookup path had to do: answer in under 300 milliseconds at the ninety-fifth percentile for authenticated calls, keep each consignee's data inside its tenant boundary, and survive the loss of one database node without a manual step. Performance evidence came from a replay of three weeks of traffic against each candidate. Caching helped anonymous pages dramatically but left authenticated calls almost unchanged, because most of their latency came from a permission query that could not be cached safely.
The access rules ruled out more options than cost did. Any design that copies consignee records into a new store creates a second place to enforce tenant rules, and the team had already seen a cached response ignore a revoked carrier permission. The read replica option kept one set of rules and was the easiest to audit, but it made the database a single point of contention. The mesh option would have added service identities for dozens of internal calls, which the six-person team could not maintain without a platform group they did not have.
Actions taken
- Build a replay harness that feeds three weeks of anonymized lookup traffic into each candidate, and record p95 latency, error rate, and monthly hosting cost in one shared sheet.
- Move the permission check for authenticated lookups into a short, cached token that carries tenant identifiers and expires quickly, and add a test that a revoked carrier loses access within one minute.
- Add a read replica dedicated to tracking lookups, route only the authenticated read path to it, and set a replication lag alarm that disables replica reads when lag exceeds a fixed number of seconds.
- Put the edge cache in front of public tracking pages only, with a short lifetime and a purge step tied to the shipment update job.
Results
The replay showed the edge cache cut anonymous page latency sharply, and the replica change brought authenticated lookups back under the target at peak. Hosting cost rose modestly rather than doubling, which the finance group accepted. The team did not build the service mesh. Two carriers received a written note explaining the change and the new lookup figures, and neither moved integrations. Six months later the team is still the same size, and its runbook fits on four pages.
Questions for discussion
- Which latency gains in your own system would survive a replay of real traffic, and which exist only in a benchmark run on quiet data?
- When a cache would make a page faster, who checks that it does not also reveal something it should not?
- How large would your team need to grow before a service mesh would stop being the most expensive option on your list?
Case Study 5
Evaluating an automated abuse triage proposal
Kestrel Bay Forums, a fictional community platform for hobbyist discussion boards, with a paid trust and safety team of seven people.
The situation
Kestrel Bay's trust and safety team reviews about 1,400 abuse reports each week, and the backlog had grown to nearly two days during a busy stretch. A vendor offered a model that would read each report, attach a risk label, and close low-risk reports automatically. The pitch focused on speed: the vendor's summary said review time could fall by half. The team asked for a pilot on past reports before committing, and the vendor supplied a hosted endpoint with access to the report text and the reported account's metadata, including region and account age.
The challenge
The trust and safety lead, Dariusz, was not against automation. His concern was that a model trained on old decisions would reproduce whatever those decisions had got wrong, and that closing reports quietly would remove the audit trail the team used to find its own mistakes. The vendor's documentation did not say how the model had been evaluated across regions, languages, or account types, and the pilot data had no field recording who had made each original decision.
Analysis
Evaluation started with metrics that matched the actual harm. Speed was the vendor's headline, but the team needed to know the rate of wrongly closed reports, measured separately for each region and language group, and the rate of legitimate accounts flagged for suspension. The pilot ran against 18,000 past reports whose outcomes had been confirmed on appeal. Overall agreement looked strong, at roughly nine in ten, but the disagreement was concentrated in reports written in languages other than English, where the model closed a larger share of valid complaints. A single accuracy number would have hidden that.
Access and maintainability raised the risks that the pilot could not measure. The endpoint would receive report text and account metadata, data that the privacy terms limit to internal reviewers, so the contract needed a processing addendum and a training restriction first. Maintainability was the other concern: a vendor-hosted model can change behavior when the vendor updates it, which would alter outcomes without any change on the team's side. The team asked for version pinning and a rollback path, and wrote a rule that no model change goes live without a rerun of the regional evaluation.
Actions taken
- Define the release gate in writing: closed-report error rate for each region and language group must stay within a set margin of human decisions, and any group outside the margin blocks rollout.
- Run the model in shadow mode for four weeks, logging its label beside the human decision without acting on it, and review every disagreement in the closed-report categories with a named reviewer.
- Keep a human in the loop for every suspension: the model may suggest a flag, but only a trained reviewer can apply it after reading the full report.
- Pin the vendor model version, keep the previous version available for rollback, and log every automated closure with the model version, so an audit can reconstruct any decision.
Results
The pilot was stopped at the shadow stage. The regional gap held up in a second review, and the vendor agreed to retrain on a sample that included the underrepresented languages, but the team did not switch on automatic closure. Instead, the model now drafts a suggested label that reviewers accept or change, and average review time fell by about a fifth. Reviewers now also sample ten percent of drafted labels each week. The backlog is shorter, and the audit trail is intact.
Questions for discussion
- Which error rate would your team accept for a group that rarely appears in pilot data, and who should make that call?
- If a vendor model changes overnight, what in your own process would reveal the change before users did?
- How would you decide whether a suggested label helps reviewers or simply makes them agree faster with the model?
Learning Resources
Technical CASE STUDIES
China, India, United States, Indonesia, Pakistan, Nigeria, Brazil, Bangladesh, Russia, Japan, Mexico, Philippines, Vietnam, Egypt, Turkey, Germany, United Kingdom, France, Italy, Canada, South Korea, Australia, Spain.