vaayaa Inclusive Education
Technical risk assessment

Large Language Models Risk Assessment: Data Exposure, Access and Control Gaps

A worked example of how a small team that runs a large language model finds, scores and controls risks to confidential data, access keys, output quality and service availability, with owners, warning signs and review triggers.

Updated November 2026 8 hazards assessed Review every 6 months Highest risk: High to Medium
See the Large Language Models course

Executive summary

This assessment shows how a small team that runs a hosted large language model can find, rate and control the risks that matter most to its data, access and service. Eight risks were chosen because they are plausible in everyday assistant work. Prompt injection, data leakage through prompts and logs, and compromised access keys carry the highest scores before controls. Each risk is scored before controls and again after existing and added controls. Most risks fall to Medium or Low once the added controls are in place, though prompt injection and key compromise stay at Medium because the impact remains serious. The first actions are a masking filter for prompts, a secrets vault for keys, weekly reply sampling and pinned versions. The team is a teaching example, so every score illustrates the method, not a real organization.

Focus area
Large language model deployment and security
Setting
Internal assistant and customer support tool built on a hosted large language model, used by a mid-sized service business
Method
Five by five likelihood and impact risk register

Scope

This assessment covers a hypothetical team that connects a hosted large language model to an internal assistant, a customer-facing chat feature, prompt and response logs, and stored company documents. It looks at confidentiality, integrity and availability of data, access to the model, third-party dependency, privacy duties, logging and incident response. It does not test any real system or provider. Jurisdiction, provider terms and technical architecture are not specified, so they are treated as labeled assumptions.

Context

The team has a product owner, two developers, an IT administrator and a data protection lead. Users type requests into a web interface, which sends them with selected company documents to the hosted model using an application programming interface key. Responses are stored in a log for quality review. Each risk is scored before controls to show its natural level, then again after existing controls and the additional controls that should be added, so students can see how treatment changes the score.

Objectives, assets and stakeholders

The objective is to deliver a useful assistant while protecting confidential data, keeping access limited to authorized people, meeting privacy duties and keeping the service available. The assets in scope are customer and company data sent through prompts, stored logs, the provider account and its keys, the application code and its libraries, and the trust customers place in the replies. Stakeholders include the product owner, developers, the IT administrator, the data protection lead, customers, staff who use the tool, the provider and the relevant regulator. Every risk has one owner role, and the product owner keeps the register current and reports to leadership each quarter.

Risk matrix: before and after controls

Each cell shows hazard numbers placed by likelihood and impact. Compare the two grids to see how controls move each hazard.

Before controls

Inherent risk

Risk before controls: rows are likelihood from almost certain to rare, columns are impact from insignificant to catastrophic. Cells list hazard numbers.
1
Insignificant		 2
Minor		 3
Moderate		 4
Major		 5
Catastrophic
5 Almost certain
4 Likely Likelihood 4, impact 3, score 12, high: H05 Likelihood 4, impact 4, score 16, high: H01 H02
3 Possible Likelihood 3, impact 3, score 9, medium: H06 H07 Likelihood 3, impact 4, score 12, high: H04 H08 Likelihood 3, impact 5, score 15, high: H03
2 Unlikely
1 Rare

After controls

Residual risk

Risk after controls: rows are likelihood from almost certain to rare, columns are impact from insignificant to catastrophic. Cells list hazard numbers.
1
Insignificant		 2
Minor		 3
Moderate		 4
Major		 5
Catastrophic
5 Almost certain
4 Likely
3 Possible Likelihood 3, impact 3, score 9, medium: H01
2 Unlikely Likelihood 2, impact 2, score 4, low: H06 H07 Likelihood 2, impact 3, score 6, medium: H02 H04 H05 H08 Likelihood 2, impact 4, score 8, medium: H03
1 Rare
  • Low 1 to 4
  • Medium 5 to 9
  • High 10 to 16
  • Critical 17 to 25

Hazard register

A summary first, then the detail for each hazard.

Summary of hazards with scores before and after controls
No.HazardBeforeAfterOwner
H01 Prompt injection and manipulated instructions 16 High 9 Medium Product owner
H02 Confidential data leaking through prompts and logs 16 High 6 Medium Data protection lead
H03 Phishing, malware and compromised access keys 15 High 8 Medium IT administrator
H04 Third-party provider dependency and supply chain exposure 12 High 6 Medium Procurement and supplier manager
H05 Inaccurate or harmful output relied on for decisions 12 High 6 Medium Service operations lead
H06 Privacy and data protection duties not met 9 Medium 4 Low Data protection lead
H07 Service outage and slow recovery 9 Medium 4 Low Operations manager
H08 Insider misuse and gaps in logging and incident response 12 High 6 Medium Security lead
H01

Prompt injection and manipulated instructions

Technology and data Before 16 After 9

Users, or content in retrieved documents, hide instructions that override the system rules, causing the model to reveal internal text, ignore policy or take actions the team did not intend.

Who or what is affected: Customers, staff users and the wider business

Causes

Free text input, untrusted documents added to prompts and no clear separation between instructions and data.

Consequences

Disclosure of internal instructions, harmful replies, misused tools and loss of customer trust.

Existing controls

  • The system instructions limit the assistant to support topics.
  • Replies are shown in a panel with no direct actions on company systems.

Control gaps

Input and retrieved content are not tested against known injection patterns, and tool permissions are broader than needed.

Additional controls

  • Prevent: Separate trusted instructions from user and document content and restrict tools to read only actions.
  • Detect: Run a test suite of injection attempts before each release and review flagged replies weekly.
Likelihood 4 x impact 4 before; 3 x 3 after Owner: Product owner Target: Within 30 days Early warning: Replies that quote internal instructions or refuse normal requests after unusual input.
H02

Confidential data leaking through prompts and logs

Technology and data Before 16 After 6

Staff paste customer records, contracts or credentials into prompts, and logged prompts and replies keep that data in places with weaker access control than the source system.

Who or what is affected: Customers, employees and the data protection lead

Causes

Staff unsure what may be shared, long log retention and no data classification at the point of entry.

Consequences

Privacy breach, contractual breach, customer complaints and a duty to report to regulators where applicable.

Existing controls

  • Staff training covers confidential data at induction.
  • Logs are stored in the company cloud account with access limited to the team.

Control gaps

Logs are kept for a long period with the same access as general staff, and no filter checks inputs before they are sent.

Additional controls

  • Prevent: Mask personal identifiers and secrets in prompts with an automatic filter before sending.
  • Detect: Scan logs weekly for patterns such as card numbers, email addresses and key formats.
Likelihood 4 x impact 4 before; 2 x 3 after Owner: Data protection lead Target: Within 30 days Early warning: Logs containing identifiers or secret-like strings in the weekly scan.
H03

Phishing, malware and compromised access keys

Technology and data Before 15 After 8

Attackers trick staff into giving up sign in details, or malware on a developer laptop reads stored keys, which then allow unauthorized use of the model and large unexpected usage charges.

Who or what is affected: Developers, the IT administrator and the finance team that pays usage bills

Causes

Phishing emails, reused passwords, and keys pasted into code or shared chat channels.

Consequences

Unauthorized use, data taken through the model, unexpected usage costs and suspension of the service.

Existing controls

  • Multi factor sign in is enabled for the provider account.
  • Keys are stored in a password manager rather than in code.

Control gaps

Keys have no spending limit and no regular rotation schedule.

Additional controls

  • Prevent: Store keys in a secrets vault, rotate them every quarter and set a monthly spending cap.
  • Respond: Revoke any key found in a repository or showing unusual use at once and issue a new key.
Likelihood 3 x impact 5 before; 2 x 4 after Owner: IT administrator Target: Within 30 days Early warning: Usage spikes or requests from unknown network locations on the provider account.
H04

Third-party provider dependency and supply chain exposure

Supplier Before 12 After 6

The provider changes model versions, pricing, data terms or hosting region without enough notice, or a software library in the application chain is compromised, and the team cannot respond quickly.

Who or what is affected: Product owner, customers and the finance team

Causes

Single provider, no tested exit plan and reliance on many open source packages.

Consequences

Broken features, forced rework, unexpected data transfers and rising costs.

Existing controls

  • The contract includes a notice period for material changes.
  • Dependencies are listed in a manifest that is reviewed at each release.

Control gaps

Provider change notices are not monitored and package versions are not pinned.

Additional controls

  • Prevent: Pin model and package versions and test upgrades in a staging environment first.
  • Detect: Subscribe the product owner to provider notices and run a dependency vulnerability scan each month.
Likelihood 3 x impact 4 before; 2 x 3 after Owner: Procurement and supplier manager Target: Within 60 days Early warning: Provider notice of a version change, or a package alert describing a known weakness.
H05

Inaccurate or harmful output relied on for decisions

Operational Before 12 After 6

The model produces plausible but wrong answers, invented sources or biased advice, and staff or customers act on them without checking, causing poor decisions and customer harm.

Who or what is affected: Customers, staff and people who make decisions from replies

Causes

Model limits, missing context, out of date documents and pressure to answer quickly.

Consequences

Wrong advice, refunds, customer harm and loss of trust in the service.

Existing controls

  • The interface states that answers may be wrong.
  • Staff are told not to send replies to customers without checking them.

Control gaps

Answers are not sampled, and the checks rely on individual judgment without a standard.

Additional controls

  • Prevent: Ground answers in approved documents and show a source reference beside each factual claim.
  • Detect: Sample a set number of replies each week and score them against a written checklist.
Likelihood 4 x impact 3 before; 2 x 3 after Owner: Service operations lead Target: Within 45 days Early warning: Rising complaints or corrections of assistant replies in the weekly sample.
H06

Privacy and data protection duties not met

Legal and compliance Before 9 After 4

Personal data is sent to the provider, stored or reused without a lawful basis, clear notice, a retention limit, or a way for people to ask for their data to be corrected or deleted.

Who or what is affected: Customers, employees and the relevant regulator

Causes

Unclear data map, no record of processing for the assistant, and provider terms not checked against team practice.

Consequences

Complaints, regulator inquiry, penalties where applicable and forced deletion of stored data.

Existing controls

  • A privacy notice is published on the website.
  • The provider contract includes a data processing clause.

Control gaps

The record of processing does not cover the assistant, and no retention period has been set for logs.

Additional controls

  • Prevent: Update the record of processing, set a retention period for logs and turn off provider reuse of company data.
  • Respond: Set a process to answer access and deletion requests within any time limit that applies.
Likelihood 3 x impact 3 before; 2 x 2 after Owner: Data protection lead Target: Within 60 days Early warning: A deletion or access request that cannot be traced to stored prompts.
H07

Service outage and slow recovery

Operational Before 9 After 4

The provider or the internal gateway fails, rate limits are reached during peak use, or a faulty release breaks the assistant, so staff and customers cannot get help for hours.

Who or what is affected: Customers, support staff and the operations team

Causes

Single hosting region, no fallback route and peak demand without limits.

Consequences

Lost sales, a backlog in support, customer frustration and extra manual work.

Existing controls

  • The provider status page is checked each working day.
  • Releases are tested in a staging area before going live.

Control gaps

There is no documented fallback to human support and no recovery steps have been rehearsed.

Additional controls

  • Prevent: Set rate limits and a request queue for peak use, and keep a second provider option ready.
  • Recover: Keep a manual support script and restore the last working release within a set time.
Likelihood 3 x impact 3 before; 2 x 2 after Owner: Operations manager Target: Within 45 days Early warning: Error rates above the agreed level for more than 15 minutes.
H08

Insider misuse and gaps in logging and incident response

People Before 12 After 6

An employee or contractor misuses the assistant to extract data, bypass rules or reach other teams' files, and weak logs or no incident plan mean the event is noticed late and not contained.

Who or what is affected: Customers, colleagues and the security lead

Causes

Broad access rights, leavers whose access stays active, and logs that are not reviewed for unusual patterns.

Consequences

Data loss, delayed containment, unclear evidence and duties to report breaches.

Existing controls

  • Access is removed when staff leave, using a written checklist.
  • Each user has an individual account that records activity.

Control gaps

There is no incident playbook for the assistant, and logs are not reviewed for access patterns.

Additional controls

  • Prevent: Apply least privilege roles and review access rights each quarter.
  • Respond: Maintain an incident playbook naming the first responder, evidence steps and reporting contacts, and test it twice a year.
Likelihood 3 x impact 4 before; 2 x 3 after Owner: Security lead Target: Within 60 days Early warning: One user accessing many customer records in a short period.

Rating method and evaluation

Each risk is rated on two five-point scales. Likelihood runs from Rare to Almost certain, and impact runs from Insignificant to Catastrophic. Multiplying the two gives a score from 1 to 25, which falls into one of four bands: Low from 1 to 4, Medium from 5 to 9, High from 10 to 16 and Critical from 17 to 25. Inherent risk is the score before any controls. Residual risk is the score after the existing and added controls, and it must never be higher than the inherent score. Impact is judged on data exposed, customers affected, cost, time to recover and harm to trust. Anything still High needs senior attention and a dated plan, anything Medium needs routine controls with a named owner, and anything Low is accepted and watched. Staff who use the tool should challenge the ratings, and every rating is reviewed when a monitoring indicator reaches its threshold or the provider changes the model.

Likelihood, impact and rating bands

Likelihood

  • 5 Almost certain: Expected to happen in most circumstances.
  • 4 Likely: Will probably happen in many circumstances.
  • 3 Possible: Could happen at some time.
  • 2 Unlikely: Not expected, but it could happen.
  • 1 Rare: Only in exceptional circumstances.

Impact

  • 1 Insignificant: No lasting effect. Handled within routine work.
  • 2 Minor: Small effect that is quickly corrected.
  • 3 Moderate: Noticeable effect that needs managed effort to correct.
  • 4 Major: Serious effect on people, delivery or reputation.
  • 5 Catastrophic: Severe or lasting effect that threatens the whole activity.

Bands (likelihood x impact)

  • Low (1 to 4): Accept and monitor.
  • Medium (5 to 9): Manage with routine controls.
  • High (10 to 16): Senior attention and a treatment plan.
  • Critical (17 to 25): Act first. Do not proceed until reduced.

Monitoring, escalation and review

Review this assessment every 6 months, and sooner when an indicator below reaches its threshold.

Early-warning indicators and escalation thresholds
Indicator to watchEscalate when
Prompts or logs containing identifiers or secret-like strings in the weekly scan.Escalate to the data protection lead the same day, and to the security lead if any key is exposed.
Usage spikes or access from unknown locations on the provider account.Escalate to the IT administrator at once, and revoke the key if the spike cannot be explained within one hour.
Sampled replies with errors above the agreed threshold in a week.Escalate to the service operations lead when errors exceed the threshold, and pause the feature if they persist for two weeks.
Provider notices of material change, or package alerts describing a known weakness.Escalate to the procurement and supplier manager within two working days, and to the product owner before any upgrade.
Error rates above the agreed service level for more than 15 minutes.Escalate to the operations manager at once, and to the executive sponsor if the outage lasts over one hour.

Contingency arrangements

If a serious event occurs, such as exposed keys, leaked customer data or a prompt attack that reveals internal instructions, the IT administrator revokes affected keys and the security lead isolates the feature. The data protection lead preserves logs, assesses what was disclosed and informs the product owner. Where a reporting duty applies, the legal adviser decides the route and timing. Service is restored from the last working release, and staff use the manual support script in the meantime. Customers are told what they need to know plainly and early.

Prioritized action plan

  1. Mask personal data and secrets in prompts with an automatic filter, owner data protection lead, within 30 days.
  2. Move keys into a secrets vault, rotate them and set spending caps, owner IT administrator, within 30 days.
  3. Ground answers in approved documents and sample replies weekly, owner service operations lead, within 45 days.
  4. Pin model and package versions and test upgrades in staging first, owner procurement and supplier manager, within 60 days.
  5. Write and rehearse an incident playbook that covers the assistant, owner security lead, within 60 days.

Conclusion

Safe use of a large language model depends on a few steady habits: keeping sensitive data out of prompts, protecting keys, grounding answers in approved sources, tracking provider changes and rehearsing incidents. Prompt injection, data leakage and key compromise carry the highest scores before controls, and their residual scores fall once the added controls are in place. Treat the register as a living record, review it on the stated cycle or after any provider change, and seek qualified legal, security and privacy advice for any real deployment.

Assumptions and limitations

Assumptions

  • The provider contract covers data handling, but the exact terms are not reviewed in this example.
  • Jurisdiction is not specified, so privacy and legal duties are described in general terms only.
  • The team uses managed devices, single sign on and a shared code repository with access controls.
  • Model replies are advisory, and a person checks them before any customer or financial decision.

Limitations

  • Scores are examples for learning and do not describe any real service, provider or organization.
  • Model behavior changes with provider updates, so ratings need review whenever the provider changes the model.
  • This is educational content and does not replace a formal site-specific assessment or licensed legal, security or privacy advice, and specialist review is needed for any real deployment.

Related frameworks to study

  • Risk register with likelihood and impact scoring
  • Hierarchy of controls
  • Three lines model of risk ownership
  • Confidentiality, integrity and availability security triad
  • Secure software development life cycle practices

Technical Risk Assessment

China, India, United States, Indonesia, Pakistan, Nigeria, Brazil, Bangladesh, Russia, Japan, Mexico, Philippines, Vietnam, Egypt, Turkey, Germany, United Kingdom, France, Italy, Canada, South Korea, Australia, Spain.

For Learning Reference Only: This assessment demonstrates how risk assessment principles are applied in a practical setting, providing a useful reference for learning and understanding. Actual assessments may be adapted to suit specific site conditions and circumstances.

Keep learning

Study the subject behind this assessment.

Large Language Models course Read the best practices for this course

More Technical risk assessments

Hourly Active: 45,552 Learners