Site Reliability Engineering

01Purpose and scope

• This manual supports the course Site Reliability Engineering. It is written as a working reference for engineers, operators, and technical leads who own or support production services and want a shared way to decide how reliable those services need to be, how much change they can absorb, and how problems get handled when they occur.
• Site Reliability Engineering, usually shortened to SRE, treats operations as a software problem. The core activities are defining reliability targets, measuring service behavior against those targets, reducing repetitive manual work, responding to incidents, and feeding what is learned back into design and staffing. The discipline overlaps with DevOps, platform engineering, and traditional systems administration, and organizations draw the boundaries differently. This manual describes practice in general terms and does not prescribe one org chart.
• The scope covers request-driven services, batch pipelines, and internal platforms. It does not cover hardware procurement, physical security, or the business decisions about pricing and contracts that sometimes depend on reliability commitments. Where a service handles personal, financial, health, or otherwise regulated data, the operational steps here are not a substitute for the controls set by the owning compliance function or for the judgment of qualified professionals in that area.

02Core principles

• Reliability is a property users experience, so targets start from user-visible behavior. A checkout API that returns errors for two percent of requests is unreliable to the people trying to buy, even if every host reports healthy. SRE work usually begins by identifying the few user journeys that matter most and choosing measurements that reflect them.
• Service level indicators, or SLIs, are the measurements. Typical examples include the proportion of requests that succeed, the proportion that complete under a latency threshold, and the proportion of data pipeline runs that finish on time. Service level objectives, or SLOs, are targets set against those measurements over a window, such as ninety-nine point nine percent of requests succeeding across a rolling thirty-day period. The exact numbers are a business and engineering choice. Higher targets cost more in redundancy, staffing, and slower change, so an SLO should reflect what users and the business actually need.
• An error budget is the allowed unreliability implied by the SLO. If the objective is ninety-nine point nine percent success, the budget is the remaining one-tenth of one percent of requests. The budget gives product and engineering teams a shared number for trade-offs: while budget remains, teams can ship risky changes; when it is spent, the priority shifts toward stability work until the service recovers. Error budgets work best when the consequences are agreed in advance and applied consistently rather than negotiated during each incident.
• Toil is repetitive, manual, automatable operational work that scales with service size and produces no lasting improvement. Restarting a stuck process every morning is toil. Writing a script that detects and restarts the process safely is engineering. SRE teams usually cap the share of time spent on toil so that automation work continues to get attention.

03Planning and preparation

• Before changing anything, establish the service inventory and the owners. Many reliability problems come from unclear ownership: a dependency that nobody monitors, an alert that routes to a retired team, or a database that three services share without an agreed support path. Record for each service its purpose, its upstream and downstream dependencies, its data classification, its on-call rota, and the channels used during incidents.
• Next, select SLIs for each critical user journey. Start with a small number, often two to four per service, covering availability, latency, and correctness where relevant. Measurement should come from the user side when possible, such as load balancer logs, client telemetry, or synthetic probes, because internal server metrics can look good while users fail. Document how each SLI is computed, including what counts as a valid request and which excluded traffic such as health checks or known bot traffic.
• Set initial SLOs using historical data where it exists, then negotiate them with product owners and stakeholders. A reasonable first target is one that the service already meets with some margin, so that the error budget conveys information rather than triggering constant alarms. Avoid setting targets from aspiration alone. Plan a review date for each SLO so it does not remain untouched after the service changes.
• Prepare the operating environment: dashboards that show SLI status and budget burn, alerts tied to user impact rather than every internal symptom, runbooks for the most frequent failure modes, and an incident channel convention. Confirm that the on-call engineer can reach the people needed for escalation and that access to production tools is granted through the normal approval process. Access requests should not be bypassed in the name of speed.

04Recommended workflow

1. 1
   : Identify the critical user journeys for the service and write one sentence for each describing what success looks like to a user. Confirm the list with the product owner and the service's main consumers.
2. 2
   : Define one to four SLIs per journey, document the calculation, and confirm that the data source is reliable enough to use. If the source has known gaps, record them and decide whether to fix the instrumentation before setting targets.
3. 3
   : Propose SLOs with a rolling window and agree on them with the owners. Record the error budget that follows and the policy that applies when the budget is consumed quickly or fully.
4. 4
   : Build or refine alerts. Page a human only when user-facing budget burn is fast enough to threaten the objective within a meaningful window. Use multi-window burn rate alerts where the monitoring system supports them, and send slower burns to a ticket queue for work during business hours.
5. 5
   : Write or update runbooks for the top failure modes. Each runbook should state the symptom, the likely cause, the first safe action, the verification step, and the condition for escalation. Keep runbooks short enough to use under pressure.
6. 6
   : Run the incident process when an alert or user report indicates impact. Assign an incident commander to coordinate, a communications lead to update stakeholders, and one or more responders to diagnose and mitigate. Prioritize restoring service over finding root cause during the incident.
7. 7
   : Once service is stable, write a blameless postmortem covering timeline, impact, contributing factors, what went well, and actions with named owners and due dates. Review the postmortem with the affected teams and track action completion.
8. 8
   : Review error budget status at a regular cadence, such as a weekly reliability review. Decide whether to continue feature launches, slow them, or prioritize specific reliability work, and record the decision.
9. 9
   : Measure toil and automate the highest-cost items first. Schedule automation work alongside feature work and make it visible in the team's planning backlog so it is not displaced indefinitely.

05Quality checks, risk controls and compliance

• Changes to production are the most common source of incidents in many environments, so SRE practice usually includes staged rollouts, automated tests, canary deployments, and the ability to roll back quickly. Each control should be proportionate to the risk. A static content change does not need the same gate as a change to an authorization service. Document the rollback procedure for every deployable component and test it occasionally, since a rollback that has never been exercised often fails when needed.
• Capacity planning is a quality check as well. Track utilization, saturation, and headroom for the resources the service depends on, and set alerts before exhaustion rather than at it. Load testing against realistic traffic patterns helps, but it has limits: synthetic tests rarely reproduce every production dependency or every failure mode, so treat results as evidence rather than proof.
• Disaster recovery and backup procedures need periodic exercises. A backup that has not been restored is an assumption, not a control. Record recovery time and recovery point objectives where the business has set them, and test against them on a schedule agreed with the owners.
• Regulatory and contractual obligations vary by jurisdiction, sector, and customer agreement. Data retention, breach notification, audit logging, change approval, and availability reporting may all carry formal requirements. This manual cannot state what those requirements are for a particular organization. Consult the current authoritative source in your jurisdiction, the organization's compliance or legal function, and the contract terms for each customer. Where a regulator or auditor has specified a procedure, that procedure controls over any general SRE practice described here.

06Collaboration and documentation

• SRE work sits between product development and operations, so the working relationships matter as much as the tooling. Product teams need to understand why a burned error budget changes their plans. Support and customer success teams need accurate status and expected timelines. Security and compliance teams need to be involved early when a change affects access control, data flow, or logging. Agree on who decides what before an incident, not during one.
• Documentation should be kept close to the work and reviewed on a schedule. The useful set is typically the service catalog entry, SLO definitions with the calculation details, dashboards with links in the catalog, runbooks, incident records, postmortems, and the reliability review notes. Each document should name an owner. Stale runbooks are a common cause of slower recovery, so include a review date and a check that the referenced commands and dashboards still exist.
• Postmortems should be written in a blameless style. The purpose is to understand how the system and the process allowed the failure, not to find a person to hold responsible. Blameless does not mean without accountability. Action items need owners, deadlines, and a way to confirm completion. Publish postmortems widely inside the organization so that other teams can learn from them.
• Communication during incidents should follow a consistent pattern: what is affected, what users see, what is being done, and when the next update will come. Avoid speculation about cause in external updates until it is verified. Internal channels should be kept for working detail so that stakeholders are not forced to read a stream of diagnostic messages.

07Measures and continuous improvement

• Track a small set of indicators that show whether reliability practice is working. Useful examples include the percentage of time each SLO is met, the rate at which error budgets are consumed, the number of incidents per service over a period, time to detect and time to mitigate, the proportion of alerts that were actionable, and the share of engineering time spent on toil. None of these is a complete picture, and each can be gamed. For example, a team that raises SLO targets to make budgets look healthy may have simply moved the goalposts. Compare trends rather than single values and read them alongside the incident records.
• Alert quality is a measurable problem. Track how many pages result in no action, how many alerts fire during planned maintenance, and how many incidents were detected by users before monitoring caught them. Each of these points to a specific fix, such as tuning a threshold, adding a maintenance window, or adding a missing probe.
• Improvement work should be scheduled and tracked like product work. Assign postmortem actions to sprints or planning cycles, and review the completion rate. When the same failure mode appears twice, treat the second occurrence as evidence that the first action was insufficient and reopen the analysis.
• Targets and practices should change as the service and its users change. Revisit SLOs when a service is re-architected, when a major customer's needs change, or when the error budget has been either never used or constantly exhausted. A budget that is never consumed may mean the target is too loose or that the team is not shipping enough. A budget that is always exhausted suggests the target is too tight for the current architecture.

08Common mistakes and how to prevent them

• Setting SLOs to one hundred percent. A perfect target leaves no room for planned maintenance, deployment risk, or the failures of dependencies outside the team's control. It also makes the error budget zero from the start, which removes its value as a decision tool. Choose targets that reflect real user tolerance and the cost of higher reliability.
• Measuring only internal health. CPU, memory, and process status can look normal while users cannot complete their tasks. Prevent this by anchoring SLIs in user-facing signals and by keeping at least one probe that exercises the full path a user takes.
• Alerting on every symptom. Paging engineers for each anomaly produces fatigue and slower response to real incidents. Page on user impact and route informational signals to dashboards or tickets.
• Treating runbooks as archives. A runbook that describes a system that no longer exists will mislead responders under stress. Attach runbook review to changes in the service and test the steps during game days or planned drills, where appropriate and safe.
• Turning postmortems into blame sessions. When people fear consequences, they hide information, and the next incident is harder to diagnose. Keep the focus on system conditions and decisions, and ask what made the wrong action look reasonable at the time.
• Letting automation languish. Teams sometimes build a script, then nobody owns it, it breaks, and the manual process returns. Assign ownership, test automation like production code, and include it in the service documentation.
• Ignoring the error budget policy. A policy that is written and never applied loses credibility quickly. If the budget is exhausted and launches continue without comment, the team learns that the target does not matter. Make the consequences explicit and follow them, or revise the policy openly.

09Practical checklist

• Confirm the service owner, on-call rota, and escalation path are recorded in the service catalog.
• List the critical user journeys and write a one-sentence success definition for each.
• Document one to four SLIs per journey with the exact calculation and data source.
• Agree SLOs with product owners and record the rolling window and error budget.
• Write the error budget policy, including the actions taken when the budget is fast-burning or exhausted.
• Configure burn-rate alerts that page only for user-threatening impact and route slower burns to tickets.
• Confirm dashboards show SLI status, budget remaining, and recent deployments.
• Write or update runbooks for the top failure modes and assign a review date.
• Verify rollback procedures for each deployable component and test them on a schedule.
• Confirm backup restores are tested against the agreed recovery objectives.
• Check with compliance or legal on retention, audit, breach notification, and customer reporting obligations for this service.
• Hold a weekly reliability review to decide on launches, reliability work, and toil reduction.
• Write blameless postmortems for significant incidents with owned, dated action items.
• Track toil hours and schedule automation work into the planning backlog.
• Review SLOs and the error budget policy after architecture changes or sustained budget patterns.

★Learning sources

• Micro Degree: Site Reliability Engineering
• Micro Degree: Data Analytics Applications
• Micro Degree: Azure Cloud Fundamentals
• Micro Degree: Google Cloud Platform Basics