Chapter 5 — Security literacy
Lesson map
Lesson 05 published. Map, example, EX013.
flowchart TD
N0["Learning outcomes"]
N1["5.1 CIA for AI"]
N2["5.2 Vocabulary"]
N3["5.3 Frameworks"]
N4["5.4 Authorised practice"]
N5["5.4b Reporting a finding"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX013"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["5.1 CIA for AI"]
N2["5.2 Vocabulary"]
N3["5.3 Frameworks"]
N4["5.4 Authorised practice"]
N5["5.4b Reporting a finding"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX013"]
[Learning outcomes]
|
v
[5.1 CIA for AI]
|
v
[5.2 Vocabulary]
|
v
[5.3 Frameworks]
|
v
[5.4 Authorised practice]
|
v
[5.4b Reporting a finding]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX013]
Learning outcomes
You will (1) use confidentiality, integrity and availability to describe AI risks; (2) read threat, vulnerability and risk correctly; (3) know the main AI-security frameworks; and (4) explain authorised practice.
5.1 CIA for AI
Confidentiality: training data, prompts, system prompts and customer records must not leak. Integrity: model outputs and actions must not be manipulated — by injection, poisoning or adversarial inputs. Availability: services must not be exhausted or disabled. NIST AI 100-2 E2025 uses similar goals (availability, integrity, privacy) to classify attacks. For Ask Brightwell, the three goals map onto three tests: whether one customer can see another's order, whether planted text can make the assistant start a return, and whether a flood of requests can exhaust the token budget. Each later chapter adds a control for one of them.
5.2 Vocabulary
A threat is a possible cause of harm (a fraudster). A vulnerability is a weakness (the assistant can be told to start returns for others). Risk combines likelihood and impact. A control reduces risk. Being precise helps teams agree on what to fix. In Brightwell's case, the cross-customer order lookup was a vulnerability; a fraudster wanting refunds is the threat; the policy service is the control. Keeping these words apart stops a team from calling the fraudster the vulnerability or the model the threat, which confuses who should act. It also helps when writing a finding report, because each part has a different reader.
5.3 Frameworks
NIST AI RMF (AI 100-1) organises risk work into Govern, Map, Measure, Manage. The OWASP Top 10 for LLM Applications 2025 lists common application risks from LLM01 Prompt Injection to LLM10 Unbounded Consumption. MITRE ATLAS catalogues attacker techniques against AI. ISO/IEC 42001 defines an AI management system. Jonas uses them together for Brightwell: the OWASP list to name application risks in Ask Brightwell, ATLAS for attacks on ReturnGuard such as extraction, and the AI RMF's functions to organise the work. ISO/IEC 42001 becomes relevant if Brightwell seeks certification for its AI management.
5.4 Authorised practice
Only test systems you are authorised to test, in writing, within scope. Use isolated copies and synthetic data. Report findings through agreed channels. Curiosity is not authorisation. Jonas's authorisation covers the Ask Brightwell staging copy, not production. When he wanted to check whether the cross-customer lookup also worked live, he asked his manager instead of trying, and the test was run on staging with test accounts. Every later test in this book stays inside the same written scope. Written scope also protects Jonas if a staging test breaks something, because the test was agreed in advance.
5.4b Reporting a finding
A good finding report states the system and version, the steps to reproduce on the authorised copy, the observed effect, the impact in CIA terms, a severity, and a suggested fix. Keep it factual and avoid exploit details beyond what the fixing team needs. Jonas's report on the cross-customer lookup followed this shape: staging version, the request "show order 100245" from a different test customer, the address shown, a confidentiality impact, and the fix in the tool service. His report left out any wording that would help misuse it.
5.5 Worked example (synthetic Brightwell case)
Jonas lists Ask Brightwell risks in CIA terms: confidentiality — order details of other customers; integrity — a planted review instructs the assistant to approve returns; availability — token floods. He maps the second to OWASP LLM01 and LLM06 (excessive agency) and checks his test authorisation covers the staging copy only. Writing the risks this way gave each a test he could run on staging and a control to propose. Confirming the scope of his authorisation before testing kept the work on the staging copy, where planted reviews and test customers can do no harm.
5.6 Errors and limits
- Calling every issue a "vulnerability".
- Testing production "just to check".
- Using frameworks without mapping them to the system.
Jonas's list for Ask Brightwell avoided these. The token-flood risk is described as a design gap rather than a vulnerability in the model, every test stayed on staging, and each framework entry is tied to a part of the drawing. A label that cannot be traced to a component tells nobody what to fix. These habits make his later reports easy for developers to act on.
Exercises
- EX013 — CIA. Give one confidentiality, integrity and availability risk for ReturnGuard. Rubric: 3 = three apt; 2 = two; 1 = one; 0 = none.
- EX014 — Vocabulary. Label threat, vulnerability, risk and control in a short scenario. Rubric: 3 = all four correct; 2 = three; 1 = one or two; 0 = none.
- EX015 — Authorisation. A friend asks you to test their company's chatbot. What must exist first? Rubric: 3 = written authorisation from owner, scope, rules, reporting channel; 2 = three; 1 = one; 0 = just do it.
Synthesis
Security literacy gives shared words — CIA, threat, vulnerability, risk, control — shared frameworks, and the discipline of authorised practice.
Continue studying — chapter 05 of 30
← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →