Data Science and Business Analytics Case Studies
Technical5 Case StudiesUpdated Sep 2026
Five cases follow, each drawn from work that data science and business analytics teams handle once models and dashboards leave the notebook. They cover a production incident, a reporting dispute traced to bad source data, a suspected credential compromise, a scaling decision for an analytics service, and an automated scoring tool that promised speed and introduced new errors. The organizations are fictional. Each case returns to the same four concerns: what the system must do, what the performance evidence shows, who can reach it, and whether the people who inherit it can keep it running.
Case Study 1
Pricing service latency after a carrier promotion
Tidewater Freight Exchange, a regional freight brokerage that runs pricing and rate scoring for about 1,200 carriers
The situation
Tidewater runs a pricing service that scores freight lanes and returns rate suggestions to the brokerage desk. On a Friday in the second week of the quarter, the team pushed a release that changed how the service joined carrier history to live lane data. Traffic rose the following Monday, when a carrier promotion sent about three times the usual request volume to the scoring endpoint. Within the hour, median response time climbed from under 300 milliseconds to more than four seconds, and about one request in twelve began timing out. Dispatchers started quoting rates by phone. The on-call engineer, Dario, found that the join had lost its index after a schema change, so every request scanned the history table.
The challenge
The service had to be restored before the next morning's rate sheet, and the team had no clean rollback because the release also carried a migration that had already run. Tidewater's rules required that any change to access rights on the history store be approved by the data platform lead, and Dario could not grant himself the read role needed to examine query plans. Leadership wanted to know whether customer data had been exposed by the new join. The team had to answer that question without slowing the investigation.
Analysis
Teams grounded in Data Science and Business Analytics usually start an incident by writing down what the service must do under load, not by reading the latest code change. Tidewater's runbook set the requirement at a response time under one second at the 95th percentile during the promotion window. Latency rose at the same minute the release reached production, and the query plan for the scoring endpoint had changed from an index seek to a full scan. Dario compared the plans from before and after the deployment and checked the request mix, so the team would not blame the release for a traffic effect alone.
The access question needed its own path. Tidewater's read role on the history store was held by three people, and the change that created the join had also widened a service account to read the raw customer table. Reducing that account was the fastest way to limit exposure, and it could be done without touching the scoring logic. The maintainability problem was older than the release. Schema changes had been applied by hand, with no migration check that tested whether indexes survived. Adding that check, and writing the index dependency into the table's documentation, made the next change easier to review than the last one.
- 1Freeze the releaseStop further deployments and confirm which schema change reached production first.
- 2Restore the indexRebuild the lost index in a low traffic window, then compare query plans.
- 3Narrow the service accountRemove raw table read rights, keeping only the aggregate view.
- 4Add migration checksTest that indexes survive each schema change before the change reaches production.
- 5Review the incidentWrite the timeline, record who accessed what, and assign owners to each follow up.
Actions taken
- Dario froze further deployments and asked the platform lead to approve a temporary grant of read access, logged against his ticket, so he could compare plans without widening rights permanently.
- The index was rebuilt in a low traffic window, then the same query set ran against a production copy to confirm the plan had returned to an index seek.
- The service account lost direct read rights on the raw customer table and was given the aggregate view instead, with each change recorded in the access log and reviewed by the data platform lead.
- The exposure review checked access logs for the four days the join ran, found no reads outside the service account, and the findings went into the incident record with named owners.
Results
Service returned to its one second target within the hour after the index rebuild, and the response time at the 95th percentile stayed under that line through the rest of the promotion. Timeouts dropped to nearly zero. The access review found no reads by accounts outside the service, so leadership had a documented answer for the customer question. The migration check is still in review. The team now runs the query plan comparison before every schema change, which turned a guess during an incident into a routine check.
Questions for discussion
- Which of the requirements in the runbook would have caught the index loss before the promotion, and where should that check live?
- What evidence would have been enough to justify a temporary grant of read access, and who should have approved it?
- How would you decide whether a schema change needs a performance test before release, and who owns that decision?
Case Study 2
Reconciling three conflicting revenue reports before the board pack
Brightwater Retail Group, a multi-region apparel retailer with about 340 stores and a central finance analytics team
The situation
Brightwater's finance analytics team produces a monthly revenue summary that feeds the board pack and the store operations review. In the first week of the month, the summary showed a revenue total for the western region that was about four percent higher than the figure in the regional dashboard, and the merchandising team had a third number from its own extract. Each figure came from a different path. The summary read a nightly warehouse table, the dashboard read a cached copy, and the merchandising extract read the point of sale feed directly. Nobody had written down which of them was authoritative, and the board pack was due before the end of the week.
The challenge
The pack could not go out with three revenue numbers, but the team did not yet know whether the gap was a timing difference, a missing store, a currency conversion, or a bug in one of the pipelines. Yusuf, the analytics engineer on the close, had access to the warehouse but not to the point of sale vendor's export logs, which were held by the store systems group. The deadline was short and the reviewers wanted a single figure with a stated basis, not a list of possible causes.
Analysis
Brightwater's board pack needed one revenue figure per region, reconciled to the ledger within a small tolerance, with the basis stated on the cover page. That requirement set the test. A pipeline could be fast and still wrong, so the team compared row counts and totals at each handoff: point of sale to staging, staging to warehouse, warehouse to dashboard cache. The totals matched until the staging step, where the western region lost about two hundred transactions from stores that had switched to a new terminal firmware in the prior month. The row count check caught the loss in minutes, faster than any reviewer scanning the summary.
Access made it worse. Three teams could write to the shared staging schema, and the merchandising extract had a direct connection to the point of sale feed left over from a one-off project. Limiting writes to the pipeline service account, and moving the extract to read from the warehouse, removed two paths where a figure could change without review. The transformation that dropped the transactions was a hand-edited filter that only one engineer understood, kept in a notebook rather than version control. Moving it into tested code with a named owner, and adding a daily row count alert, meant the next firmware change would show up as an alert rather than a board question.
- First morningSummary and dashboard totals differed by about four percent.
- Same morningRow counts compared at each handoff, gap isolated to staging.
- Within two hoursTerminal firmware change matched the stores with missing transactions.
- Second dayFilter rewritten as tested code and reconciliation alert added.
- Before the pack deadlineRestated totals reconciled to the ledger with the basis stated on the cover.
Actions taken
- Yusuf agreed a single owner for the western region figure and asked the store systems group for the vendor export logs, which confirmed the firmware timing within the same afternoon.
- The missing transactions were reloaded from the vendor export into staging under a tagged batch, so every restated row could be traced back to its source file and load date.
- The hand-edited filter was replaced with a function under version control, reviewed by a second engineer, with a test that fails if a store code is dropped.
- The dashboard now reads from the warehouse rather than its cache, and the summary footer shows the row count, the ledger tie out, and the date of the last successful load.
Results
The board pack went out with one western region figure, tied to the ledger within the agreed tolerance and carrying a basis note that named the reloaded batch. Store operations began checking terminal firmware before rollouts, and the merchandising team retired its direct feed. In the following month the daily alert fired twice, both times for an upstream delay that was real, and each was cleared before the summary ran. Reviewers asked for the same footer on the other close reports, and the finance analytics team agreed to add it in the next cycle.
Questions for discussion
- Which single source should the board pack have treated as authoritative, and how would that have been written down before the close began?
- What would a reasonable tolerance look like for a reconciliation alert on row counts, and who should decide it?
- If the vendor export had not been available, what would you have reported to reviewers, and how would you have labeled it?
Case Study 3
Suspicious logins on a shared analytics service account
Larkspur Insurance Services, a mid-sized regional insurer whose analytics group builds claims models for underwriting
The situation
Larkspur's analytics group runs a shared service account that reads claims extracts for scheduled model training. In the second week of the month, the security monitoring team flagged the account for logins from a hosting provider address range that no analyst used. The logins succeeded, and the account then listed several storage buckets it had not touched in months. The account had a long-lived key that had been copied into two notebooks and a scheduler configuration, and nobody could say which of those copies were still in use. The monitoring team opened an incident and asked the analytics lead, Tomasz, to avoid deleting anything until evidence was preserved.
The challenge
The team had to decide how far to contain before they knew the scope. Shutting the account off would stop the attacker but would also break the nightly training job that two underwriting models depended on. Rotating the key was quick, but if the logins came through a copy still in a scheduler, the rotation would fail silently. Ayesha, the security engineer on call, also needed to preserve the sign-in records before any retention policy removed them.
Analysis
In a security case the performance question turns into an evidence question. The analytics group required that model training finish before the morning review, which meant the job history had to show which credential ran each step. The logs supported that only partly. The identity provider recorded every sign-in, but the storage platform logged reads only for buckets with access logging turned on, and two of the buckets the account touched had it off. Ayesha pulled the sign-in records first, then matched timestamps against the training job history, which separated the suspicious activity from the scheduled runs.
Access was the root of the exposure. A single key with broad read rights, shared by a service account, meant that any copy of the key was as powerful as the original. The first fix was to replace the shared key with a workload identity bound to the scheduler, scoped to the three buckets the training job read, and to require that any new grant be approved by the data platform owner. Maintainability was the second fix. The notebooks that held copies were stale and undocumented, so the team inventoried them, removed the embedded credentials, and added a check that fails the build when a secret appears in a repository.
Actions taken
- Ayesha exported sign-in records, job history, and bucket access logs to a sealed evidence store with hashes recorded, before any retention rule could remove them.
- The shared key was disabled after a workload identity was tested in the scheduler, so the nightly training job kept running on a new credential that could read only the three buckets it needed.
- Access to the two buckets with logging disabled was limited to the security team, and logging was turned on for every bucket holding claims data, with alerts for reads from outside known ranges.
- The inventory covered every notebook, scheduler entry, and repository that had touched the key, and each copy was either removed or assigned an owner with a review date.
Results
Containment finished within the working day, and the review found that the attacker had listed buckets and read one extract of claims summaries before the key was disabled. Larkspur notified the data protection officer, who handled the reporting obligations under its internal procedure. The training job ran on its new identity that night. Eleven copies of the key were found and removed, and the build check now blocks new secrets from reaching a repository. The incident record lists every action with its time, owner, and evidence hash.
Questions for discussion
- Which of the logs that were switched off would have changed the scope finding, and what should have been turned on before the incident?
- When a shared credential supports a nightly job, how would you schedule its replacement so the job is not the thing that fails?
- Who should decide that a containment step is authorized, and how should that approval be recorded while the incident is still open?
Case Study 4
Choosing a serving design for a growing customer analytics API
Kestrel Commerce Platform, a marketplace software vendor whose analytics API serves dashboards to about 4,000 merchant accounts
The situation
Kestrel's analytics API serves merchant dashboards with daily sales, returns, and conversion figures. Over the last several months, merchant count grew by about a third, and the API began to miss its latency target during the afternoon peak, when most merchants refreshed their dashboards. Two designs were on the table. The first kept the existing aggregation on demand, with a larger database cluster. The second precomputed daily rollups into a separate store and served dashboards from it. Costs and risk differed between them, and the platform team had to recommend one before the next budget review.
The challenge
The platform team had a target of 800 milliseconds at the 95th percentile for dashboard loads, and the current figure was above two seconds at peak. The on-demand design was cheaper to change but scaled with query volume, which was growing fastest. The rollup design was faster but meant a second store to secure, a nightly job that could fail, and a data freshness promise to merchants. Ines, the platform architect, had to show how each option would hold up under the access rules that applied to merchant data.
Analysis
The requirements decide which trade-offs matter. Merchants need figures that are no more than fifteen minutes old, a dashboard load under 800 milliseconds at the 95th percentile, and the ability to filter by any of about forty fields. Load tests on the on-demand design showed the database CPU saturating at about three times current peak, with the slowest queries spending most of their time in a join on returns. The rollup prototype cleared the latency target, but its freshness depended on a nightly job finishing within a window the team had measured only once. The evidence favored rollups on speed, yet it was thin on the failure path.
Access rules shape the design more than the diagrams suggest. Merchant data is isolated by tenant, and the rollup store would hold aggregated figures for every merchant in one place, so it needs row level policies that match the ones on the source tables. Ines proposed keeping the rollup store inside the same account boundary and issuing read credentials per tenant, which keeps the audit trail intact but adds a policy layer that the team must test. On maintainability the case is less clear. The rollup path adds a job, a schema, a backfill procedure, and an owner for each. A scripted backfill and a freshness alert kept it manageable.
- Measure load Replay peak traffic against both designs and record latency and database saturation.
- Check access Confirm tenant isolation and credential scope on every new read path.
- Cost the failure Price the cost of a missed freshness window, not only the monthly infrastructure bill.
- Assign ownership Name who is paged when the rollup job fails and who approves schema changes.
Actions taken
- Ines ran the replay of peak traffic against both designs over a full week of logged requests, which exposed the join pattern that drove database saturation.
- The team built the rollup prototype behind a feature flag, so merchants on a pilot list saw the new store while everyone else stayed on the on-demand path.
- Read credentials were scoped per tenant, and the access review covered every new policy before the pilot widened, with a test that fails if any tenant can read another's rows.
- The nightly rollup job was given a paging alert, a documented backfill script, and a freshness dashboard that merchants could see, so a late run was visible before it became a complaint.
Results
The team chose the rollup design for the dashboard path and kept the on-demand query path for ad hoc filters that the rollups did not cover. Afternoon dashboard loads came in under the latency target at peak, and the database cluster ran at a fraction of its previous load. The pilot surfaced one real problem, a backfill that ran twice on a retried job and double counted returns for a single day, which the team fixed with an idempotent write before wider rollout. Infrastructure cost rose, and the team accepted that for the freshness alert and tenant isolation.
Questions for discussion
- Which measurements would you need before you trusted a freshness window that had been observed only once, and how often would you repeat them?
- How should the team price a failure mode such as a missed nightly job when comparing it with a higher monthly infrastructure bill?
- What signals would make you reverse the decision after the next several months, and which should be watched from the start?
Case Study 5
Evaluating an automated claims triage model before rollout
Corvin Benefits Administration, a third-party administrator that processes about 9,000 reimbursement claims each week
The situation
Corvin's operations group processes reimbursement claims for several employer plans. Claims come in as scanned forms and structured submissions, and a reviewer currently reads each one before it is paid or sent back. A vendor proposed an automated triage model that would classify each claim as clear, needs review, or likely invalid, and route it accordingly. The vendor's summary reported a large reduction in handling time and a high overall accuracy figure from its own test set. The operations director, Marisol, wanted to know whether the model could be trusted with routing and where it could fail.
The challenge
The claims that matter most are the ones the model is least likely to see clearly, such as handwritten forms, provider codes that were revised midcycle, and claims from smaller employers whose plans had little history in the training data. A false clear sends money out on an invalid claim, and a false invalid delays a member's reimbursement and invites a complaint. Ahmed, the claims quality lead, needed a way to measure both errors separately, across the populations the model would actually handle, before anyone agreed to a pilot.
Analysis
Start with the decision the metric supports. Corvin's routing needs a false clear rate and a false invalid rate, measured separately for each plan type and claim format, plus time to decision from receipt. An overall accuracy figure hides both problems, because a model that handles printed claims well can average out a failure on handwriting. Ahmed asked the vendor to run the model on a sample of 1,600 past claims, stratified by format and employer size, and to report the confusion matrix for each stratum. The results showed handwritten claims had a false clear rate several times the printed rate, which the summary had not mentioned.
Access and maintainability matter here because a triage model changes who sees what. The pilot would send claims with member health details to a vendor hosted service, so the data processing terms, the encryption at rest, and the list of staff who can export outputs all needed review. Corvin's policy allowed the vendor to read claim images only for the pilot period, with deletion confirmed in writing. The model also needed a change path. Every vendor update had to be logged, tested against the held out sample, and approved by the claims quality lead before it reached live routing, because silent changes would move the error rates unseen.
Actions taken
- Ahmed defined the acceptance thresholds before the pilot began: false clear rate under half a percent for each stratum, with no stratum allowed to fall below the human baseline.
- The vendor agreed to a pilot limited to printed claims, with handwritten forms kept on the manual path until the stratum met the thresholds in two consecutive monthly samples.
- Every routed decision was logged with the model version, the confidence score, the reviewer who handled exceptions, and the reason code, and reviewers could override any automated routing.
- Staff exports were limited to two named analysts, the model's training copy of claim images was deleted at the end of the pilot, and a monthly review checked the errors reported by members.
Results
The pilot ran on printed claims first. The false clear rate stayed under the ceiling in each monthly sample, but the handwritten stratum did not meet its threshold, so those claims stayed on the manual path. Routing time for printed claims fell by about a third, and reviewers spent their hours on the complex files the model flagged. Two defects surfaced: a provider code mapping that the model misread, and a confidence score that ran high on a form template with a common scanning fault. Both were reported and fixed in a later version approved through the same process.
Questions for discussion
- Which stratum would you test first if the vendor could share only one further breakdown, and why?
- How should a team decide the size of the weekly spot check for high confidence claims, and what would change that size?
- What evidence would convince a claims quality lead that a handwritten stratum is ready for automation, and who else should sign off?
Learning Resources
Technical CASE STUDIES
China, India, United States, Indonesia, Pakistan, Nigeria, Brazil, Bangladesh, Russia, Japan, Mexico, Philippines, Vietnam, Egypt, Turkey, Germany, United Kingdom, France, Italy, Canada, South Korea, Australia, Spain.