OpenCyber AcademyLearning outcomes
You will (1) use confidentiality, integrity and availability to describe AI risks; (2) distinguish threat, vulnerability, risk and control; (3) name the main AI-security frameworks and what each is for; (4) explain why AI systems need specific attention; and (5) apply these ideas to a service you know.
1.1 The CIA triad applied to AI
Confidentiality: customer records, uploaded documents, system prompts and training data must not reach the wrong people. Integrity: answers, scores and actions must not be manipulated — for example by text hidden in an uploaded document. Availability: the service must stay up and affordable; floods of long requests can exhaust budgets. NIST AI 100-2 E2025 uses comparable attacker goals (availability, integrity, privacy) to organise attacks on AI.
For ClaimChat, all three apply at once, which is why Sofia starts her assessment with them.
1.2 Basic vocabulary
A threat is a possible source of harm (a fraudster). A vulnerability is a weakness (ClaimChat books callbacks for any claim number). Risk combines how likely a harm is with how bad it would be. A control reduces risk (an ownership check). Precise words help teams agree quickly on what to fix.
In Sofia's example, the callback weakness becomes a high risk only because a fraudster could use it against many customers' claims. Use the four words consistently in findings, so readers know whether you mean a weakness, a harm or a fix.
1.3 Frameworks and their uses
- NIST AI RMF (AI 100-1): organising risk work into Govern, Map, Measure and Manage.
- NIST AI 600-1: risks specific to generative AI.
- OWASP Top 10 for LLM Applications 2025: the most common LLM application risks, from LLM01 Prompt Injection to LLM10 Unbounded Consumption.
- MITRE ATLAS: attacker tactics and techniques against AI, with case studies.
- ISO/IEC 42001: an AI management system.
- EU AI Act (Regulation 2024/1689): legal obligations by risk level.
Sofia uses OWASP to name ClaimChat's risks and the AI RMF to organise her work.
1.4 Why AI needs specific attention
AI systems behave statistically; they can be steered by text an attacker supplies (prompt injection); they learn from data that can be corrupted; and their outputs can leak training data. Ordinary controls still matter — access control, logging, patching — but AI adds new paths to harm.
At Meridian, ClaimChat reads uploaded repair quotes written by people outside the company, and FraudFlag learns from labels set by investigators. Both are paths an ordinary web application does not have, so Sofia's assessment covers them alongside access control and logging.
1.5 Worked example (synthetic Meridian case)
On her first day, Sofia describes ClaimChat in CIA terms. Confidentiality: a customer might see another customer's claim. Integrity: a repair quote containing hidden text might make ClaimChat tell a customer their claim is approved. Availability: automated scripts could flood ClaimChat during a storm, when real customers need it most. She maps the second to OWASP LLM01 and the first to LLM02.
Each scenario names a harm, the people affected and a framework entry, so the team can discuss them without ambiguity.
1.6 Errors and limits
- Treating AI as a black box outside normal security.
- Confusing a vulnerability with a risk.
- Quoting frameworks without applying them.
Each error causes trouble later. Treating AI as outside normal security leaves basics such as access control and logging unchecked around ClaimChat. Confusing a vulnerability with a risk leads teams to argue about severity without agreeing what the weakness is. Quoting a framework entry without saying how it applies to the system, as Sofia does for LLM01 and LLM02, gives the reader nothing to act on. Check your first assessment notes for all three.
Exercises
- EX001 — CIA. Write one confidentiality, integrity and availability risk for FraudFlag. Rubric: 3 = three apt; 2 = two; 1 = one; 0 = none.
- EX002 — Vocabulary. Label threat, vulnerability, risk and control for the callback scenario. Rubric: 3 = four correct; 2 = three; 1 = one or two; 0 = none.
- EX003 — Frameworks. Which framework would you use to (a) list LLM app risks, (b) organise a risk programme, (c) study attacker techniques? Rubric: 3 = OWASP, AI RMF, ATLAS; 2 = two; 1 = one; 0 = none.
- EX004 — Why AI. Give three reasons AI needs specific attention. Rubric: 3 = three distinct; 2 = two; 1 = one; 0 = none.
- EX005 — Apply. Describe an AI service you know in CIA terms. Rubric: 3 = three specific risks; 2 = two; 1 = one; 0 = none.
Synthesis
Practitioners start with shared vocabulary and frameworks, then apply them to the specific ways AI services can be steered, corrupted or exhausted.