vaayaa Inclusive Education
Technical risk assessment

Site Reliability Engineering Risk Assessment: Access, Change, Recovery and Vendor Risks

A worked example of how a site reliability team finds, scores and controls the risks that threaten service availability, access, customer data and recovery, from identity compromise and misconfiguration to vendor failure and on-call fatigue.

Updated November 2026 8 hazards assessed Review every 6 months Highest risk: Critical to Medium
See the Site Reliability Engineering course

Executive summary

This assessment shows how a site reliability team can find, rate and control the risks that threaten a live service and the customer data it holds. Eight risks were chosen because they are plausible in everyday operation, from identity compromise and misconfiguration to failed releases, vendor outages, data exposure and on-call fatigue. Malware, phishing and insider misuse carry the highest inherent score because a single successful attempt can reach production data and source code. Each risk is scored before controls and again after existing and added controls. Most fall to Medium or Low once multi-factor sign in, least privilege access, progressive releases, tested restores and centralized logging are in place, but malware and insider exposure keep residual impact high enough to need senior attention. The first actions are hardware backed multi-factor sign in, a timed restore drill, progressive rollout gates and a single incident log. The service is a teaching example, so every score illustrates method, not a real organization.

Focus area
Site reliability engineering and service operations
Setting
Operations team running a customer-facing online service on hosted infrastructure
Method
Five by five likelihood and impact risk register

Scope

This assessment covers the production service, its hosting environment, deployment pipelines, monitoring and logging, on-call operations, incident response and the vendors the team depends on. It looks at risks to availability, confidentiality, integrity, recovery and privacy duties. It does not cover the commercial terms of any real contract or the security posture of any real organization. Jurisdiction, scale and tooling are assumptions made for teaching and are labelled as such in the assumptions list.

Context

The example service is a customer-facing web application with an API, a database holding customer account records, and a set of background jobs. It runs on hosted cloud infrastructure managed through infrastructure as code. A small on-call rota of engineers handles pages, a service owner sets reliability targets, and a platform team maintains shared tooling. Changes reach production through automated pipelines with peer review. Each risk is scored before controls to show its natural level, then again after existing controls and the additional controls proposed here, so students can see how the rating moves.

Objectives, assets and stakeholders

The objective is to keep the service available within agreed reliability targets, protect customer data from unauthorized access or change, recover quickly from failure, and meet the privacy and contractual duties the team owes to customers. The assets in scope are the production service and its APIs, the customer account database and backups, deployment pipelines and infrastructure configuration, monitoring and log stores, access credentials, and the on-call runbooks. Stakeholders include the on-call engineers, the service owner, the platform team, customers, the security and privacy contacts, third-party vendors and senior leadership. Every risk has one owner role, and the service owner reviews the register with the team each month and reports material changes to leadership.

Risk matrix: before and after controls

Each cell shows hazard numbers placed by likelihood and impact. Compare the two grids to see how controls move each hazard.

Before controls

Inherent risk

Risk before controls: rows are likelihood from almost certain to rare, columns are impact from insignificant to catastrophic. Cells list hazard numbers.
1
Insignificant		 2
Minor		 3
Moderate		 4
Major		 5
Catastrophic
5 Almost certain
4 Likely Likelihood 4, impact 3, score 12, high: H02 Likelihood 4, impact 4, score 16, high: H01 H03 Likelihood 4, impact 5, score 20, critical: H05
3 Possible Likelihood 3, impact 2, score 6, medium: H08 Likelihood 3, impact 4, score 12, high: H04 H06
2 Unlikely Likelihood 2, impact 5, score 10, high: H07
1 Rare

After controls

Residual risk

Risk after controls: rows are likelihood from almost certain to rare, columns are impact from insignificant to catastrophic. Cells list hazard numbers.
1
Insignificant		 2
Minor		 3
Moderate		 4
Major		 5
Catastrophic
5 Almost certain
4 Likely
3 Possible Likelihood 3, impact 3, score 9, medium: H03 H06
2 Unlikely Likelihood 2, impact 2, score 4, low: H01 H08 Likelihood 2, impact 3, score 6, medium: H02 H04 Likelihood 2, impact 4, score 8, medium: H05
1 Rare Likelihood 1, impact 5, score 5, medium: H07
  • Low 1 to 4
  • Medium 5 to 9
  • High 10 to 16
  • Critical 17 to 25

Hazard register

A summary first, then the detail for each hazard.

Summary of hazards with scores before and after controls
No.HazardBeforeAfterOwner
H01 Unauthorized access and identity compromise 16 High 4 Low Platform team lead
H02 Misconfiguration and unpatched vulnerabilities 12 High 6 Medium Platform team lead
H03 Failed deployments and slow recovery 16 High 9 Medium Release engineering lead
H04 Slow detection and weak incident response 12 High 6 Medium On-call coordinator
H05 Malware, phishing and insider misuse 20 Critical 8 Medium Security lead
H06 Third-party and supply chain dependency 12 High 9 Medium Vendor manager
H07 Customer data confidentiality, integrity and privacy 10 High 5 Medium Data protection lead
H08 On-call fatigue and key person dependency 6 Medium 4 Low On-call coordinator
H01

Unauthorized access and identity compromise

Technology and data Before 16 After 4

Stolen or shared credentials, excessive standing permissions and weak session controls let an attacker or former staff member reach production systems, change configuration or read customer records.

Who or what is affected: Customers, on-call engineers and the platform team

Causes

Reused passwords, long lived tokens, shared break glass accounts and departing staff whose access is not removed.

Consequences

Customer data theft, service tampering, regulatory notice and loss of trust.

Existing controls

  • Single sign on is used for internal tools.
  • Production access is granted through a ticket that a manager approves.

Control gaps

Break glass accounts are shared, and access reviews happen only once a year.

Additional controls

  • Prevent: Enforce hardware backed multi-factor sign in and replace standing production rights with time limited access.
  • Detect: Alert on new privileged sign ins, unusual locations and access granted outside working hours.
Likelihood 4 x impact 4 before; 2 x 2 after Owner: Platform team lead Target: Within 30 days Early warning: Production access that has not been used in the last 30 days but remains active.
H02

Misconfiguration and unpatched vulnerabilities

Technology and data Before 12 After 6

Infrastructure changes, open network rules, outdated packages and default settings expose services or data that were meant to stay private, often without anyone noticing until a scan or incident reveals them.

Who or what is affected: Customers, the platform team and the wider engineering group

Causes

Manual console edits, copied templates with default settings and patches that wait for a convenient window.

Consequences

Exposed storage, unauthorized access, forced emergency changes and outages during patching.

Existing controls

  • Infrastructure is defined in version controlled code.
  • Dependency updates are proposed by an automated tool each week.

Control gaps

Console changes can still bypass code review, and findings have no fixed deadline.

Additional controls

  • Prevent: Scan configuration against agreed secure baselines before every change and block changes that fail.
  • Detect: Run regular external and internal scans and track open findings to a fixed closing date.
Likelihood 4 x impact 3 before; 2 x 3 after Owner: Platform team lead Target: Within 60 days Early warning: Configuration drift detected between the deployed state and the version controlled definition.
H03

Failed deployments and slow recovery

Operational Before 16 After 9

A release that has not been tested against real traffic, or a database migration that cannot be reversed, causes a wide outage, and rollback takes longer than the agreed recovery time because steps are manual or undocumented.

Who or what is affected: Customers, the on-call engineers and the service owner

Causes

Large batch releases, schema changes made without a reverse step and little load testing.

Consequences

Extended outage, lost transactions, customer churn and breach of reliability targets.

Existing controls

  • Releases pass an automated test suite before merge.
  • Each deployment has a named engineer who can roll back.

Control gaps

Rollback has never been timed end to end and load tests use synthetic data only.

Additional controls

  • Prevent: Use progressive rollouts with automatic halt rules tied to error rate and latency targets.
  • Recover: Script and rehearse rollback and migration reversal so the steps finish within the recovery target.
Likelihood 4 x impact 4 before; 3 x 3 after Owner: Release engineering lead Target: Within 60 days Early warning: Changes that roll back more than once in a month or that exceed the error budget.
H04

Slow detection and weak incident response

Operational Before 12 After 6

Logs are incomplete, alerts fire late or not at all, and responders lack a clear command structure, so an incident is found by customers rather than by the team and evidence is lost during the response.

Who or what is affected: Customers, responders and the security contacts

Causes

Gaps in instrumentation, log retention set too short and alerts tuned to produce noise.

Consequences

Longer outages, lost forensic evidence and repeated failures.

Existing controls

  • Metrics dashboards cover the main user journeys.
  • An incident process names an incident commander for major events.

Control gaps

Log sources for background jobs are not complete and postmortem actions are not tracked.

Additional controls

  • Detect: Add alerts on user facing error rates and on missing log streams, and test them each quarter.
  • Respond: Keep a single incident log and a written command checklist, and hold a blameless review after each major event.
Likelihood 3 x impact 4 before; 2 x 3 after Owner: On-call coordinator Target: Within 45 days Early warning: Incidents first reported by customers rather than by internal alerts.
H05

Malware, phishing and insider misuse

Technology and data Before 20 After 8

Phishing emails, malicious packages or a rogue insider with legitimate access install malware, exfiltrate source code or secrets, or sabotage production, often without triggering a standard alert.

Who or what is affected: Customers, engineers, vendors and the wider organization

Causes

Convincing phishing lures, unvetted third-party packages, secrets stored in code and broad rights held by a few engineers.

Consequences

Data theft, sabotage, ransom demands and emergency rebuilds.

Existing controls

  • Email filtering blocks known malicious attachments.
  • Engineers complete an annual security awareness course.

Control gaps

Secrets still appear in some configuration files and build agents have wide outbound access.

Additional controls

  • Prevent: Store secrets in a managed vault, rotate them on a schedule and block unsigned or unvetted packages in builds.
  • Detect: Monitor for unusual bulk downloads, new scheduled jobs and outbound connections from build and production hosts.
Likelihood 4 x impact 5 before; 2 x 4 after Owner: Security lead Target: Within 30 days Early warning: A build or production host that opens an unexpected outbound connection or installs an unlisted package.
H06

Third-party and supply chain dependency

Supplier Before 12 After 9

A hosting provider, managed database, monitoring tool or open source library fails, changes terms or is compromised, and the service inherits the outage or the vulnerability with little warning and no fast exit.

Who or what is affected: Customers, the vendor manager and the platform team

Causes

Concentration on one provider, deep dependency chains and reliance on a tool with no export path.

Consequences

Extended outage, forced migration, unpatched exposure and cost overrun.

Existing controls

  • Critical vendors have named contacts and service credits in their contracts.
  • Open source libraries are checked against a known vulnerability list before use.

Control gaps

Exit routes are described on paper but have not been tested, and the dependency inventory is incomplete.

Additional controls

  • Prevent: Pin versions, keep a vetted internal mirror of critical libraries and review vendor security evidence each year.
  • Respond: Document a failover or exit route for each critical vendor and rehearse it twice a year.
Likelihood 3 x impact 4 before; 3 x 3 after Owner: Vendor manager Target: Within 90 days Early warning: Vendor status incidents or missed service level commitments from a critical dependency.
H07

Customer data confidentiality, integrity and privacy

Legal and compliance Before 10 After 5

Customer records are copied into logs, test environments or analytics tools, or are altered by a faulty job, so personal data is exposed, kept too long or corrupted, breaching privacy commitments and duties.

Who or what is affected: Customers, privacy contacts and the data owner

Causes

Verbose logging, production copies used for testing and batch jobs that run without checks.

Consequences

Privacy breach, notification duties, possible penalties and loss of customer trust.

Existing controls

  • Personal fields are encrypted in the main database.
  • Retention periods are written into the data handling policy.

Control gaps

Logs are not reviewed for personal data and deletion jobs are not tested.

Additional controls

  • Prevent: Mask personal data in logs and non test environments and block copies of production data without approval.
  • Detect: Run checks that compare record counts and checksums after batch jobs and flag unexpected changes.
Likelihood 2 x impact 5 before; 1 x 5 after Owner: Data protection lead Target: Within 60 days Early warning: Personal identifiers found in log samples or test environments during routine checks.
H08

On-call fatigue and key person dependency

People Before 6 After 4

A small rota carries heavy paging, unclear handover and reliance on one expert for critical systems, so responders become tired, make mistakes during incidents and leave the team with knowledge gaps.

Who or what is affected: On-call engineers, the team and customers

Causes

Noisy alerts, small team size and knowledge held by one person.

Consequences

Burnout, slower response, errors in recovery and staff turnover.

Existing controls

  • A rota rule limits consecutive on-call weeks.
  • Runbooks exist for the most common alerts.

Control gaps

Handover notes are inconsistent and on-call workload is not measured.

Additional controls

  • Prevent: Set a paging budget per shift and retire alerts that do not need a human response.
  • Respond: Pair new responders with an experienced engineer and cross train on each critical system each quarter.
Likelihood 3 x impact 2 before; 2 x 2 after Owner: On-call coordinator Target: Within 90 days Early warning: Pages per on-call engineer per shift above the agreed limit for two shifts running.

Rating method and evaluation

Each risk is rated on two five-point scales. Likelihood runs from Rare to Almost certain, and impact runs from Insignificant to Catastrophic, with impact judged on customer harm, data exposure, length of outage, recovery effort, contractual or legal consequences and loss of trust. Multiplying likelihood by impact gives a score from 1 to 25, which falls into one of four bands: Low from 1 to 4, Medium from 5 to 9, High from 10 to 16 and Critical from 17 to 25. The score is a guide for ordering work, not a measurement. Inherent risk is the score before any controls are counted. Residual risk is the score after existing controls and the proposed additional controls are in place, and it must never be higher than the inherent score. Anything rated Critical or High needs senior attention and a dated plan with a named owner. Anything Medium needs routine controls and a regular check. Anything Low is accepted and watched. Ratings are challenged in a short review where an engineer outside the owning team asks what evidence supports each score, and ratings are revisited after any incident, vendor change or failed drill.

Likelihood, impact and rating bands

Likelihood

  • 5 Almost certain: Expected to happen in most circumstances.
  • 4 Likely: Will probably happen in many circumstances.
  • 3 Possible: Could happen at some time.
  • 2 Unlikely: Not expected, but it could happen.
  • 1 Rare: Only in exceptional circumstances.

Impact

  • 1 Insignificant: No lasting effect. Handled within routine work.
  • 2 Minor: Small effect that is quickly corrected.
  • 3 Moderate: Noticeable effect that needs managed effort to correct.
  • 4 Major: Serious effect on people, delivery or reputation.
  • 5 Catastrophic: Severe or lasting effect that threatens the whole activity.

Bands (likelihood x impact)

  • Low (1 to 4): Accept and monitor.
  • Medium (5 to 9): Manage with routine controls.
  • High (10 to 16): Senior attention and a treatment plan.
  • Critical (17 to 25): Act first. Do not proceed until reduced.

Monitoring, escalation and review

Review this assessment every 6 months, and sooner when an indicator below reaches its threshold.

Early-warning indicators and escalation thresholds
Indicator to watchEscalate when
Accounts or tokens with production access that have not been used in the last 30 days.Escalate to the platform team lead at once and remove the access within 7 days, with the service owner informed if the access was used through a shared login.
Production changes deployed without a passed automated check or a peer review.Escalate to the release engineering lead the same day, and to the service owner if more than one change in a month bypasses checks.
Failed restore tests or backups older than the agreed recovery point.Escalate to the service owner immediately, and to senior leadership if a restore test fails twice in a quarter.
Pages per on-call engineer per shift and alerts acknowledged after 15 minutes.Escalate to the on-call coordinator when load exceeds the agreed limit for two shifts, and to the service owner if it persists for a month.
Vendor incidents or missed service level commitments from a critical dependency.Escalate to the vendor manager within one hour, and to senior leadership if the outage exceeds the agreed recovery time.

Contingency arrangements

If a serious event occurs, such as confirmed account compromise, data exposure or a multi-hour outage, the incident commander declares the incident, freezes related deployments and isolates affected credentials and hosts. The on-call engineer preserves logs and evidence before any clean up. The service owner informs leadership, and the privacy and security contacts decide, with specialist advice, whether customers, vendors or regulators must be told and when. Service is restored from the latest tested backup or a standby region, and customers receive early, plain updates.

Prioritized action plan

  1. Enforce hardware backed multi-factor sign in and remove standing production access, owner platform team lead, within 30 days.
  2. Run a full restore drill from the latest backup and record the time taken, owner service owner, within 45 days.
  3. Gate every production change behind automated checks and progressive rollout, owner release engineering lead, within 60 days.
  4. Consolidate alerts and set paging thresholds that match the reliability targets, owner on-call coordinator, within 60 days.
  5. Review vendor dependencies and confirm failover and exit terms, owner vendor manager, within 90 days.

Conclusion

Reliable services depend on steady habits: least privilege access, tested restores, progressive releases, complete logs and a manageable on-call load. The highest scores sit with malware, phishing and insider misuse, identity compromise and misconfiguration, and each has actions that bring the residual score down. Treat the register as a living record, review it on the stated cycle and after every major incident, and seek qualified security, privacy and legal advice for any real decision.

Assumptions and limitations

Assumptions

  • The service is customer-facing and holds personal account data, but the jurisdiction is not specified, so privacy duties are described in general terms only.
  • Infrastructure is hosted with a third-party cloud provider and is managed through version-controlled configuration.
  • The team uses standard monitoring, centralized logs and a paging tool, and has agreed reliability targets for the service.

Limitations

  • The scores are examples for learning and do not describe any real service, team or incident.
  • Security, privacy and cloud compliance rules differ by country and change over time, so specialist review is needed.
  • This is educational content and does not replace a formal site-specific assessment or licensed security or legal advice.

Related frameworks to study

  • Service level objectives and error budgets
  • Principle of least privilege
  • Hierarchy of controls
  • Three lines model of risk ownership
  • Blameless postmortem practice

Technical Risk Assessment

China, India, United States, Indonesia, Pakistan, Nigeria, Brazil, Bangladesh, Russia, Japan, Mexico, Philippines, Vietnam, Egypt, Turkey, Germany, United Kingdom, France, Italy, Canada, South Korea, Australia, Spain.

For Learning Reference Only: This assessment demonstrates how risk assessment principles are applied in a practical setting, providing a useful reference for learning and understanding. Actual assessments may be adapted to suit specific site conditions and circumstances.

Keep learning

Study the subject behind this assessment.

Site Reliability Engineering course Read the best practices for this course

More Technical risk assessments

Hourly Active: 45,552 Learners