OpenCyber Academy
Professional · Chapter 2 · English
Course code: OCA-043Chapter 02: OCA-043-C002

Chapter 2 — AI and machine-learning security: service and stakeholder context

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 02 published. Map, example, EX006.

flowchart TD
  N0["Learning outcomes"]
  N1["2.1 Purpose, users, affected people"]
  N2["2.2 Stakeholders"]
  N3["2.3 A simple data-flow picture"]
  N4["2.4 AI assets"]
  N5["2.5 Which services first"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX006"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["2.1 Purpose, users, affected people"]
  N2["2.2 Stakeholders"]
  N3["2.3 A simple data-flow picture"]
  N4["2.4 AI assets"]
  N5["2.5 Which services first"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX006"]

[Learning outcomes]
    |
    v
[2.1 Purpose, users, affected people]
    |
    v
[2.2 Stakeholders]
    |
    v
[2.3 A simple data-flow picture]
    |
    v
[2.4 AI assets]
    |
    v
[2.5 Which services first]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX006]

Learning outcomes

You will (1) describe an AI service's purpose, users and affected people; (2) identify stakeholders and their concerns; (3) draw a simple data-flow picture; (4) record AI assets; and (5) judge which services need most attention.

2.1 Purpose, users, affected people

Start every assessment by asking what the service is for, who uses it, and who is affected by its outputs even without using it. ClaimChat's users are customers; affected people include claims handlers who receive callbacks and third parties named in claims.

For FraudFlag, affected people include honest claimants whose claims are delayed when flagged, even though they never see the model. Write the list of users and affected people at the top of every assessment, and return to it when choosing tests.

2.2 Stakeholders

Product owners want value and speed; operations wants stability; legal and privacy want compliance; customer service wants fewer complaints; security wants bounded harm. Talk to each early. Their concerns become requirements, and their support makes controls stick.

At Meridian, ClaimChat's product owner, operations, the privacy officer and customer service each raised a different concern with Sofia. Record what each stakeholder needs and who will approve changes, so later findings reach the right person. Ask each one what failure would worry them most.

2.3 A simple data-flow picture

Draw boxes for the customer, the chat app, the model, document reading, the claims database and the callback tool; arrows for data; dashed lines where trust changes (customer input, uploaded documents, model output to tools). This picture shows where untrusted content can steer actions.

Sofia's picture of ClaimChat showed uploaded documents flowing straight into the model's input. Keep the picture up to date as tools and data sources change, and use it to choose tests. Mark each point where text from outside Meridian enters, such as uploaded quotes, since those points need injection tests.

2.4 AI assets

List: model (vendor and version), system prompt, data sources, tools and their permissions, keys, logs. Note owner and sensitivity for each. System prompts can leak (OWASP LLM07), so they must not contain secrets.

For ClaimChat, Sofia's list includes the vendor model and version, the system prompt, the uploaded documents, the callback tool and its permissions, the vendor key and the chat logs. Review the list after each change, since a new tool or data source adds an asset and widens what an attacker can reach.

2.5 Which services first

Services that read untrusted content, can take actions, handle personal data, or affect many people deserve attention first.

"""Lab V: an AI system inventory scored for review priority (synthetic).
Each system lists exposure, autonomy, data sensitivity and whether it reads
untrusted content; mapping to OWASP LLM Top 10 (2025) IDs is illustrative."""
import json
inv = json.loads("""[
 {"name":"support-chatbot","exposure":"public","autonomy":"tools","data":"customer","untrusted_input":true},
 {"name":"invoice-ocr-model","exposure":"internal","autonomy":"none","data":"financial","untrusted_input":true},
 {"name":"code-assistant","exposure":"internal","autonomy":"suggest","data":"source","untrusted_input":true},
 {"name":"churn-predictor","exposure":"internal","autonomy":"none","data":"customer","untrusted_input":false},
 {"name":"hr-cv-screener","exposure":"internal","autonomy":"decide","data":"personal","untrusted_input":true}
]""")
E = {"public": 3, "internal": 1}; A = {"none": 0, "suggest": 1, "tools": 3, "decide": 3}
D = {"customer": 2, "financial": 2, "source": 2, "personal": 3}
for s in inv:
    s["score"] = E[s["exposure"]] + A[s["autonomy"]] + D[s["data"]] + (2 if s["untrusted_input"] else 0)
    risks = []
    if s["untrusted_input"]: risks.append("LLM01")
    if s["autonomy"] in ("tools", "decide"): risks.append("LLM06")
    if s["data"] in ("customer", "personal"): risks.append("LLM02")
    s["owasp"] = ",".join(risks) or "-"
for s in sorted(inv, key=lambda s: -s["score"]):
    print(f"{s['name']:18s} score={s['score']:2d} owasp={s['owasp']}")

Expected output:

support-chatbot    score=10 owasp=LLM01,LLM06,LLM02
hr-cv-screener     score= 9 owasp=LLM01,LLM06,LLM02
code-assistant     score= 6 owasp=LLM01
invoice-ocr-model  score= 5 owasp=LLM01
churn-predictor    score= 3 owasp=LLM02

ClaimChat reads uploaded documents, books callbacks, handles personal data and serves many customers, so it scores on all four points; FraudFlag scores on personal data and on the number of claimants affected. Use the lab's simple score to rank services, and explain the ranking to stakeholders. Start with the highest score and work down. A service that books actions, reads outside text and holds personal data deserves testing before one that only answers general questions.

2.6 Worked example (synthetic Meridian case)

Sofia interviews ClaimChat's product owner, a claims team lead and the privacy officer. The team lead worries about fake urgent callbacks flooding handlers; the privacy officer about uploaded documents containing health information. Sofia's data-flow picture shows uploaded documents entering the model's input directly — a path for indirect injection. Her asset list reveals that the vendor key is shared with a test environment.

These four findings came from conversations and a picture, before any testing. Sofia shared them with the product owner, who confirmed that nobody had considered uploads as a route for instructions.

2.7 Errors and limits

Each error narrows the view. Skipping affected people who are not users would leave out the claims handlers and named third parties at Meridian. Talking only to technical staff misses concerns such as the privacy officer's worry about health information in uploads. An asset list without owners leaves nobody responsible, as Sofia found when listing ClaimChat's assets. Check your context notes for all three. Fix any gap before testing begins.

Exercises

Synthesis

Context — purpose, people, stakeholders, flows and assets — decides what matters in AI security before any test is run.

Continue studying — chapter 02 of 12

← Chapter 01 (draft) · Course chapter index · Chapter 03 (draft) →

Worked answers

EX006. - Purpose: prioritise suspicious claims for investigation. - Users: the fraud team. - Affected: all claimants, especially those wrongly flagged and delayed.

EX007. - Product owner: speed and value. - Claims lead: handler workload. - Privacy officer: health data in uploads. - Operations: stability during storms.

EX008. - Components: customer → chat app → model ↔ document reader; model → callback tool → handlers; claims database. - Boundaries: customer messages, uploaded documents, model output to the tool.

EX009. A key used in test is exposed to more people and systems, so a leak in test compromises production. Rotation affects both, and usage cannot be told apart.

EX010. ClaimChat ranks first: it reads untrusted uploads, can act through the callback tool, and handles personal data. FraudFlag affects many people but takes no direct actions and has no untrusted text input to a model.