Google Cloud Platform Basics Case Studies
Technical5 Case StudiesUpdated Sep 2026
Each case below describes a team running production workloads on managed cloud infrastructure. The failures are ordinary: a release pushed late in the week, a join that quietly drops rows, a credential left in the wrong place, a model that routes tickets to the wrong queue. Each is read through four lenses: what the system must do, what the measurements show, who can reach what, and whether the next engineer can understand and change the result. Treat the discussion questions as prompts for a team meeting, and argue the position you disagree with.
Case Study 1
Latency collapse after a Thursday checkout release
Larkspur Ticketing, a mid-sized service that sells seats for about 300 venues and runs its checkout on managed infrastructure
The situation
After a release on a Thursday afternoon, the checkout API for Larkspur Ticketing began returning timeouts. The service ran on Cloud Run with a Cloud SQL backend. Traffic had been ramping toward a concert onsale that night, peaking at roughly 4,200 requests per minute. p95 latency rose from about 280 milliseconds to over 6 seconds within twenty minutes, and the error rate climbed past 12 percent. Support received forty-odd tickets in the first hour. The on-call engineer, Priya, rolled back the revision but latency stayed high, which pointed away from the release alone.
The challenge
The team had to restore sales before the onsale window closed, but they did not yet know whether the bottleneck was in the application, the database connection pool, or the autoscaling settings. Logs were split across services, and two engineers had Owner-level roles on the project, so changes were being made in parallel. They needed to stabilize first, then find out what happened, then make sure it would not recur without slowing the next release train to a crawl.
Analysis
Teams grounded in Google Cloud Platform Basics tend to begin with the system's stated requirements before reaching for dashboards. Larkspur had a checkout latency target written into its service level objective, but nobody had translated it into a per-request budget for the database calls. That gap mattered. Cloud Run scales instances by concurrency, so when each instance held more open connections than the Cloud SQL tier could serve, the database became the queue without anyone seeing it as one. The evidence sat in Cloud Monitoring: connection counts flat against the ceiling, CPU on the database well under half. Latency was rising because requests were waiting for connections, not computing.
The second half of the analysis concerns access and maintainability, which is where the team's own habits had contributed. Two engineers held Owner on the project, a role broader than either needed for deployments, so a hotfix to the connection pool setting went in without review while the release was still rolling back. Nobody could say afterward which change was active. Scoping deployers to Cloud Run Developer and database administration to a separate group, with changes recorded in audit logs, would have made the sequence visible. Maintainability suffers the same way: runbooks that name a dashboard rather than a query leave the next on-call person guessing.
- 1Hold the rollbackKeep the previous revision serving traffic while the connection data is checked.
- 2Lower concurrencyCut per-instance concurrency so fewer connections reach the database.
- 3Confirm the ceilingCompare open connections against the Cloud SQL limit in Cloud Monitoring.
- 4Reconstruct changesPull audit log entries to order every change made during the incident.
- 5Restore the guardrailsNarrow roles and require review before any project-level change ships.
Actions taken
- Keep the rollback in place and lower per-instance concurrency so each instance opens fewer database connections. Watch p95 latency against the checkout budget for thirty minutes before declaring recovery.
- Freeze deployments and direct all production changes through one incident channel, with a named change owner, until the timeline of changes is reconstructed from audit logs.
- Remove Owner from the two engineers and grant narrower roles, then add a policy that flags any project-level role change for review within one business day.
- Write the on-call runbook around specific queries and saved dashboard links, and test it in a rehearsal where the on-call engineer has never seen the service before.
Results
Checkout latency returned under the budget in the first hour after the rollback was held and concurrency was lowered. The error rate fell below one percent by the end of that evening, and the onsale closed with a sales count within a few percent of forecast. The audit review found the hotfix that had gone in without review, and it was reverted in the next maintenance window. Two weeks later the team ran a rehearsal, and the new runbook let an engineer from the support rotation identify the connection ceiling in under fifteen minutes.
Questions for discussion
- Which of the two engineers' changes should have been visible to the on-call person before the first rollback, and what would have made it visible?
- If the onsale had been moved to a smaller database tier, how would you have decided between lowering concurrency and buying capacity?
- What single runbook entry would have shortened the diagnosis most, and who on the team is best placed to write it?
Case Study 2
Conflicting weekly revenue reports from a store feed
Brackenridge Foods, a regional grocery distributor with nine warehouses and a weekly sales forecasting process
The situation
Weekly reports from Brackenridge Foods's operations team disagreed with the finance dashboard by a few percent on some weeks and by much more on others. Both drew from a BigQuery dataset that ingested point-of-sale files from store systems each night. Some stores sent returns as negative quantities, while others left them out entirely. A recently migrated store had been sending timestamps in local time without an offset, and a join to the calendar table dropped roughly 2,300 transactions near midnight each week. The category manager said the forecast was off, and the finance lead said the revenue figures were correct. Both were partly right.
The challenge
The analytics team could not tell which discrepancies were real business signals and which were pipeline artifacts. Each fix to one store feed changed the totals in another report, and the dashboards had no record of which transformation version produced a given number. Access was also uneven: the finance group could query the curated tables, but the operations analysts had been given raw dataset access to build their own views, so the same metric had been defined three ways. Leadership wanted a reorder decision within the week.
Analysis
Start with the requirement that the report was supposed to meet, not the query that produced it. The operations team needed weekly unit sales per store per category, reconciled to the register totals within a stated tolerance. Finance needed net revenue that matched the ledger. Those are two different definitions. Lineage tracing then showed that both reports read from the same ingested table, but one applied the returns convention and the other did not. The performance evidence was in the data itself: a row count comparison by store and day exposed the stores whose totals diverged, and the midnight drop showed up as a dip in transactions at the day boundary for only the migrated store.
The access pattern was the quiet cause of the drift. Operations analysts had raw dataset access, so their views applied filters and joins that never passed through the curated layer, and nothing recorded which version of a view a deck had used. Restricting analysts to read-only curated views, and putting the transformation logic in version-controlled SQL with named owners, would have prevented most of the divergence. Maintainability depends on the same change. A metric defined once, in one place, with a test that checks the returns convention, is easier to keep correct than three copies that agree until someone edits one.
- Week oneReconciled totals by store and day to find the diverging feeds.
- Early in week twoLineage tracing showed two definitions of returns reading one table.
- Mid week twoTimezone offset found in the migrated store feed at ingestion.
- End of week threeHistorical feed normalized and eight weeks rerun against the calendar.
- Following cycleReconciliation passed for every store and the reorder plan was revised.
Actions taken
- Write one definition for each reconciled metric, including the returns convention and the timezone rule, and store it alongside the transformation code.
- Normalize the migrated store feed to UTC at ingestion, then rerun the last eight weeks so historical totals stop shifting between reports.
- Add row count and total checks between the raw and curated tables, grouped by store and day, that fail the nightly load when a store deviates past the agreed tolerance.
- Restrict analysts to read-only curated views, move ad hoc logic into version-controlled SQL with named owners, and log the view version on every shared deck.
Results
The timezone fix and the returns rule brought the operations and finance totals within the agreed tolerance for every store in the next reporting cycle. The midnight drop disappeared, which recovered roughly 2,300 transactions a week in the migrated store's sales figures. Two stores had been reporting returns inconsistently for a longer period, and their corrected history changed the category forecast enough that the reorder plan was revised before the supplier window. The nightly check has failed twice since, both times catching a feed that had changed its column order, and each failure was resolved within the same morning.
Questions for discussion
- Which of the two metric definitions should have owned the final revenue number, and who should have been able to change it?
- How would you decide the tolerance for store-level deviations when a store has only a few weeks of history?
- If the analysts needed raw access for an unusual investigation, what process would grant it without reopening the drift?
Case Study 3
Suspicious service account activity at a dental group
Corbel Hill Dental Group, a network of twelve offices with a shared patient scheduling app
The situation
In the second week of the quarter, the security analyst, Marcus, noticed a service account that normally made a few hundred calls a day to the scheduling database had made several thousand calls from an unfamiliar IP range over a single night. Cloud Audit Logs showed the calls were reads of appointment and contact tables, and a JSON key for that account had been created nine months earlier by a contractor who no longer worked with the group. A second anomaly: a storage bucket holding nightly exports had its public access setting changed twice in the same window. Nothing had been reported by patients or staff yet.
The challenge
The group had no one whose job was incident response, and the practice manager wanted to know whether patient records had left the environment before deciding whether to notify anyone. Containment could break the scheduling app for every office on a Monday morning. The team had to decide how far to cut access without destroying the logs they would need to answer that question, and to do it under the authority of a named incident lead rather than whoever happened to be online.
Analysis
Separate what the system must keep running from what it must keep private. For Corbel Hill, scheduling had to stay available during office hours, and the audit trail had to survive any containment step. The evidence came from Cloud Audit Logs: the service account's data access entries showed the source addresses, query counts per hour, and the tables read. Those counts were the baseline that made the anomaly measurable. Without a baseline of a few hundred calls a day, several thousand would have looked like a busy Tuesday. The bucket change appeared in the admin activity log with the actor and time.
The root of the exposure was access design. A long-lived JSON key for a service account had outlived the contractor who created it, and the account could read the whole scheduling dataset. Workload identity federation or short-lived credentials would have removed the key, and narrowing the role to the two tables the app reads would have limited what a stolen key could reveal. The public setting on the export bucket was a separate control gap, since organization policy could have blocked public access. For maintainability, the team's documentation listed the key's owner as a department, not a person, so no one knew it existed when the contractor left.
Actions taken
- Disable the service account key without deleting it, after the incident lead records the approval, so the audit trail and the account's history remain intact for review.
- Export the relevant audit log entries to a locked evidence bucket with restricted access, and record a hash of each file in the incident log.
- Set the public access prevention control on the export bucket and rotate any credentials that could read it, then confirm from a separate identity that anonymous requests are refused.
- Scope the blast radius by reviewing every read the account made in the window against the list of patient tables, and brief the practice manager on findings before any notification decision.
Results
The key was disabled within the first afternoon, and the export bucket was closed before the next scheduled export ran. Log review showed the reads came from a single address range tied to a hosted virtual machine the contractor had used, and no patient records appeared in any outbound transfer. The practice manager and outside counsel reviewed the evidence and decided that notification was not required under the group's obligations. Within a month, every service account was given a named owner, key creation was blocked by policy, and the scheduling service moved to short-lived credentials.
Questions for discussion
- What evidence would have been enough to justify containing the scheduling app for all offices rather than only the single account?
- Who should hold the incident lead role in a practice of this size, and what authority should that person have on a Monday?
- Which of the maintainability gaps here would most likely recur in your own environment, and how would you find its owner?
Case Study 4
Choosing ingestion architecture for refrigeration telemetry
Tidewell Labs, a firm that monitors refrigeration units for restaurant and grocery clients across the region
The situation
Tidewell Labs' ingestion API accepted readings from about 3,400 refrigeration units, each reporting every thirty seconds. Peak ingestion reached about 9,500 readings per minute during a regional heat event, and alert latency stretched from under a minute to between four and seven minutes. Monthly cloud spend had nearly doubled across two quarters because a single always-on cluster ran at low utilization overnight. Two options were on the table: keep the current Kubernetes cluster and tune it, or move ingestion to Cloud Run with Pub/Sub and a streaming pipeline. Neither option had been measured against the same workload.
The challenge
The decision had to satisfy three constraints that pulled in different directions. Alerts had to arrive within a minute during peak load, costs had to fall without slowing the rollout of new sensor types, and the on-call rotation of four engineers could not absorb another cluster to patch. The security team also needed every device to authenticate with its own identity, and the chosen design had to make that easy to audit. A choice made on cost alone would have failed at least two of these.
Analysis
Begin with the workload shape rather than the platform. Readings arrive as small, steady messages with bursts tied to weather, and little state must survive between messages. Load tests on the existing cluster showed that CPU was idle most of the night while the ingestion pods held memory for connections that were rarely reused. Two prototype pipelines were then built against the same replayed week of traffic, one on Cloud Run with Pub/Sub buffering and one on the tuned cluster. Both were measured on the same two numbers, alert latency at the ninety-fifth percentile and cost per million readings, because earlier comparisons had used different data.
Security constraints weighed more heavily than expected. Every device needed a workload identity, and the cluster version relied on a shared service account whose key sat in a configuration file, which security review had flagged. Cloud Run gave each service its own identity and made per-device authorization a question of IAM bindings and short-lived tokens, which auditors could read directly. Maintainability favored the managed option since the team of four could patch a runtime image and move on, but Cloud Run's limits on long-lived connections meant the design needed a separate path for devices that held sessions open. The team recorded that path as the price of the choice.
- Measure Replay one shared week of traffic through every candidate design.
- Compare Score alert latency and cost per million readings side by side.
- Check access Confirm each device identity and audit trail meets security review.
- Operate Estimate patch, on-call, and rollback effort for the four-person team.
Actions taken
- Adopt Cloud Run for the ingestion service, with Pub/Sub buffering in front, and set minimum instances so alerts keep their latency target through the overnight trough.
- Give each device class its own service identity and grant it only the publish permission on its topic, so a compromised unit cannot read other devices' data.
- Keep the cluster running for the long-lived session path until its replacement passes a two-week soak test, then retire it in a planned window with a documented rollback.
- Set budget alerts at sixty and ninety percent of the monthly forecast, and review per-pipeline cost each week against readings processed rather than against total spend.
Results
The Cloud Run pipeline held alert latency under forty seconds at the ninety-fifth percentile through the heat event, and it cut monthly ingestion cost by roughly a third compared with the tuned cluster's projected spend. The migration took the team most of a quarter, mainly because the session path needed a second design. Device identities were issued one class at a time, and the first security audit after the change found no shared credentials in the ingestion path. Cost per million readings became the number leadership asked about each month, which changed the conversation from total spend to whether a new sensor type was worth its volume.
Questions for discussion
- At what volume of devices would you revisit the choice to run a cluster, and what evidence would trigger that review?
- How would you weigh a shorter alert latency target against a flat monthly cost ceiling if both could not be met?
- Which parts of the decision record would a new engineer need first, and how would you keep it current?
Case Study 5
Evaluating an automated claims routing model
Quillmoor Insurance Services, a brokerage that places commercial policies for about 220 small business clients
The situation
A vendor proposed a classifier to route incoming claim emails at Quillmoor Insurance Services to the right adjuster team. The demo sorted a sample of 500 emails in under a minute, and the vendor's sales lead projected that the intake queue, which averaged about 1,800 emails a week, could be cleared with two fewer coordinators. The model would run on a managed endpoint on Google Cloud, reading the subject, body, and attachment text. Early pilot numbers looked strong overall, but the team had not broken them down by policy type, sender region, or the language the client wrote in. The operations director wanted a decision before the next renewal cycle.
The challenge
Efficiency was easy to measure and bias was not. A routing error that sent a flood claim to the wrong team cost days, while a systematic pattern that sent non-English claims to a general queue could persist for months without anyone noticing. The team needed metrics that could catch both, a way for adjusters to override the model without friction, and rollout steps that could be reversed in an afternoon. Nobody on the team had built a model evaluation before, and the vendor's documentation reported only overall accuracy.
Analysis
Define what the routing decision must guarantee first, since accuracy alone does not say whether a misroute is acceptable. For Quillmoor, a claim with a possible injury had to reach a licensed adjuster within the same business day, whatever the model predicted. The requirement became a metric set: precision and recall per policy type, the share sent to the general queue, time to first human touch, and the override rate by adjuster. The pilot had reported one blended accuracy figure. Re-running a labeled sample of 1,200 emails with those metrics split by region and language showed a recall gap of twelve points for non-English claims, which the blended number had hidden.
Access and oversight needed the most care. Adjusters should see the model's suggested route and its reason, but the override log had to be writable only by the adjuster who made the change, so overrides could be audited without counting as model errors. Model endpoints needed their own service identity with no rights to the claims archive beyond the fields it scored, and the prompt and label set had to be version controlled so a retrained model could be compared against the one it replaced. Maintainability depends on someone owning the evaluation sample. Without a named owner refreshing it quarterly, the metrics would drift toward the easy cases.
Actions taken
- Run the model in shadow mode for one full month, with routing suggestions logged but not acted on, and compare them against adjuster decisions by segment.
- Set release gates: recall for injury-related claims and non-English claims must each meet the agreed floor before any claim is routed automatically, with a rollback owner named.
- Route only low-risk policy types automatically at first, and send every claim matching the injury keyword rules to a licensed adjuster regardless of the model's label.
- Publish a weekly review of override rates and sampled misroutes to the operations director, and pause automatic routing for any segment whose override rate rises above the agreed threshold.
Results
The shadow month showed the model matched adjusters on most routine claims but missed about one in eight non-English injury claims, which was enough to change the plan. The vendor retrained on a larger labeled set, and the second evaluation met both recall floors. Automatic routing went live for two low-risk policy types in the next quarter, and the coordinators' queue time fell by roughly a day on those claims. Overrides stayed within the agreed threshold for the first two months, and the team has since extended automatic routing one policy type at a time.
Questions for discussion
- Which segment would you have chosen as the first gate if the injury keyword rules had not existed, and why?
- How much adjuster override activity is healthy, and what would a rising override rate tell you that accuracy would not?
- Who should be allowed to approve a retrained model for production, and what record should that approval leave behind?
Learning Resources
Technical CASE STUDIES
China, India, United States, Indonesia, Pakistan, Nigeria, Brazil, Bangladesh, Russia, Japan, Mexico, Philippines, Vietnam, Egypt, Turkey, Germany, United Kingdom, France, Italy, Canada, South Korea, Australia, Spain.