Safety Norms: Site Reliability Engineering

1Purpose, Scope, and Safety Principles

These norms apply to engineers, operators, managers, and contractors who build, deploy, monitor, and maintain production services, including on-call rotations, capacity planning, change management, and automation. They also cover physical work where SRE staff rack, cable, or replace hardware in server rooms, colocation cages, and edge sites. This is educational guidance and does not replace professional authorization, formal training, approved procedures, supervision, or emergency services.

The governing principle is that people come first, then the public and customer data, then service availability. A reliability target never justifies bypassing a safety control. MANDATORY items are required by this document or by an applicable law, contract, or manufacturer instruction. RECOMMENDED items are good practice to adopt unless a documented reason says otherwise. Laws, standards, and manufacturer instructions are jurisdiction-dependent, so verify current versions with competent authorities or qualified professionals.

2Roles, Competence, and Accountability

The service owner is accountable for the service risk register, change approvals, and error budget decisions. The SRE lead maintains on-call readiness, runbooks, and training records. During a declared incident, the incident commander directs the response and may pause changes. Facilities or site management owns power, cooling, fire systems, lockout procedures, and physical access. SRE staff must not override those systems and must not work on energized electrical equipment unless they are authorized and trained for that work under local rules.

Competence is verified, not assumed. Production access is granted by role after training on the service, its runbooks, and privileged access rules. Physical tasks involving power, batteries, refrigerants, heavy lifting, or elevated work go to qualified facility staff or trained contractors.

• MANDATORY: every production change has a named author and, for high-risk changes, a separate approver.
• MANDATORY: privileged access is role-based, time-limited where feasible, and logged.
• MANDATORY: contractors complete a site safety induction before entering data halls.
• RECOMMENDED: pair new on-call engineers with an experienced shadow for at least one full rotation.

3Course-Specific Hazards and Risk Assessment

SRE hazards fall into four groups: service harm, data exposure, physical danger, and human factors. Service harm is the most common. A faulty deployment, runaway autoscaler, or retry storm can interrupt systems that people rely on for medical records, emergency dispatch, payments, or transport scheduling. Treat any service that supports life safety or essential payments as high risk regardless of its technical size.

Score each hazard by likelihood and severity on a five-point scale and record it in the risk register. Scores at or above a locally set threshold require a named mitigation owner and a review date. The risk levels below are starting points for local scoring, not fixed values.

• High: unreviewed production changes, broad administrative credentials, and logs containing personal data or secrets.
• High: on-call fatigue leading to incorrect commands, especially in the wrong environment or region.
• High: electrical shock or arc flash from power distribution units, switchgear, and damaged power cords in data halls.
• Medium to high: lithium-ion battery systems, heavy server chassis, and rack tipping during installation.
• Medium: overheating from cooling failure, fire suppression discharge, trip hazards under raised floors, and hot surfaces.
• Medium: refrigerant leaks, battery and e-waste disposal, and energy waste from idle capacity.

4Preventive Controls and Safe Work Practices

Change control is the primary preventive control. Every production change needs a documented rollback or forward-fix plan, a stated blast radius, and a verification step. Use progressive rollout such as canary releases, feature flags, and staged regions, and configure automated rollback where the platform supports it. Change freezes apply during major customer events or when the error budget is exhausted, and the service owner records any exception in writing.

Destructive actions need extra care. Commands that delete data, change routing, or alter access controls require a dry run where the tool offers one, a second-person check, and confirmation of the target environment. Operators use read-only views first and do not run ad hoc scripts against production from a personal laptop.

Physical work follows a permit-style approach. Before work in a data hall begins, the requester records the task, equipment, hazards, and required isolation. The facility authority approves power isolation and any hot work, and trained personnel apply lockout and tagout. Two people work together on rack tasks, lifts use mechanical aids, and no one works alone in a restricted area.

• MANDATORY: no production change proceeds without a tested rollback or forward-fix plan.
• MANDATORY: break-glass access to databases, networks, and secrets is recorded for review.
• MANDATORY: the facility authority issues a permit before power, battery, or cooling equipment is touched.
• MANDATORY: racks are anchored or stabilized according to current manufacturer instructions and local practice; verify those instructions before installation.
• RECOMMENDED: bound on-call shifts, use handoff notes, and require engineers who worked a long incident to rest before further production work.
• RECOMMENDED: review alert quality regularly so noisy alerts do not train responders to ignore real danger.

5Equipment, Environment, Information, and Protective Safeguards

Protective equipment depends on the task and site rules. In data halls, staff wear closed-toe footwear, safety glasses when cutting cable ties or handling sheared metal, and cut-resistant gloves when lifting chassis edges. Use electrostatic discharge wrist straps when handling boards and drives. Hearing protection is used where a local noise assessment shows it is needed. Specific PPE standards are jurisdiction-dependent, so confirm them with site safety personnel.

Equipment and environment controls include anchored racks, posted weight limits, labeled and dual power feeds where available, and no daisy-chained power strips. Temperature and humidity thresholds come from manufacturer documentation and facility design, and alarms route to people who can act on them. Keep aisles and exits clear and do not stack boxes against fire detection devices. Food and drink stay outside server areas unless site rules allow them.

Information safeguards apply to logs, tickets, and backups. Classify data, apply least privilege, mask personal data in incident tickets, keep secrets in an approved vault rather than in code or chat, and set retention limits. New telemetry that could capture personal data needs a privacy review. Security controls and reporting duties are set by the security team and applicable law, so treat this section as guidance. Hardware decommissioning uses approved data erasure and certified recyclers, and battery waste goes to licensed handlers.

6Incident Reporting, Emergency Response, and Escalation

Safety comes before service restoration. If there is smoke, a burning smell, sparking, water near power, or a suspected gas or refrigerant release, stop work, move people to a safe location, activate the site alarm or emergency procedure, and call the local emergency number. Do not attempt repairs or open enclosures. Report injuries to the facility lead immediately. Medical care comes from local emergency services, and first aid is given only by people trained to provide it.

For service incidents, declare severity using the locally defined scale, assign an incident commander, and open a shared response channel. The commander decides whether changes halt. Customer communication is factual and approved by the communications owner, and it avoids speculation about cause. A suspected data exposure goes to the security and privacy leads at once. Notification deadlines and recipients are jurisdiction-dependent and must be confirmed by counsel or the privacy office.

Near misses are reported in the same system as incidents. Examples include a bad change caught before deployment, mislabeled power cables spotted before work, and access granted to the wrong group. Reviews are blameless and focus on system conditions. Each review produces tracked action items with a named owner and a due date set by the team.

7Documentation and Training

Keep records that let another engineer reconstruct what happened. These include change records, runbook versions, access grants and revocations, on-call schedules, incident timelines, risk register entries, training completions, facility permits, inspection reports, and waste or recycling certificates. Retention periods are jurisdiction-dependent and contract-dependent, so confirm them with legal and records staff.

Training combines theory and practice. New engineers complete service onboarding, privileged access training, a site safety induction for data halls, and a supervised first on-call shift. Refreshers run on a locally set cycle. Drills include failover exercises with a scoped blast radius, tabletop simulations, and evacuation drills coordinated with facility staff. Drills must not affect production users unless the service owner approves them in advance.

• MANDATORY: records of privileged access and production changes are kept for the period required by applicable rules.
• MANDATORY: physical safety induction is completed before unescorted entry to a data hall.
• RECOMMENDED: tabletop exercises include a data exposure scenario and a power or cooling failure scenario.

8Monitoring, Review, and Continuous Improvement

Measure the health of the safety system, not just uptime. Useful indicators include change failure rate, rollback frequency, near-miss report volume, alert volume per on-call shift, overdue access reviews, overdue permit closures, failed physical inspections, and incidents caused by manual steps. Trends matter more than single values. A rising count of near misses often means reporting is improving rather than that the system is getting worse, so interpret it with care.

Review the norms after any major incident, after significant architecture or facility changes, and on a locally set review cycle. Each review checks whether controls still match the risk register, whether runbooks match the current platform, and whether responsibilities are still assigned to real people. Versioned changes to these norms are communicated to every person who must follow them.

Implementation checklist

• Confirm every production change has a named author, a separate approver for high-risk work, and a tested rollback or forward-fix plan.
• Verify that privileged access is role-based, time-limited where feasible, and logged.
• Check the error budget and change freeze status before starting any risky work.
• Use a dry run and a second-person check before any destructive production command.
• Obtain a facility permit before touching power, battery, cooling, or hot work.
• Use two people and mechanical aids for rack installation and lifting.
• Keep personal data and secrets out of tickets, chat, logs, and code.
• Stop work and call emergency services for smoke, sparks, water near power, or a refrigerant release.
• Report every incident and near miss within the locally defined window.
• Record actions, complete required training, and schedule a norms review after major changes.