Chapter 1 — AI and machine-learning security: foundations and decisions
Lesson map
Lesson 01 published. Map, example, EX001.
flowchart TD
N0["Learning outcomes"]
N1["1.1 How AI systems work, briefly"]
N2["1.2 How they fail or are attacked"]
N3["1.3 Organising the decision"]
N4["1.4 Options"]
N5["1.5 Recording the decision"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX001"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["1.1 How AI systems work, briefly"]
N2["1.2 How they fail or are attacked"]
N3["1.3 Organising the decision"]
N4["1.4 Options"]
N5["1.5 Recording the decision"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX001"]
[Learning outcomes]
|
v
[1.1 How AI systems work, briefly]
|
v
[1.2 How they fail or are attacked]
|
v
[1.3 Organising the decision]
|
v
[1.4 Options]
|
v
[1.5 Recording the decision]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX001]
Learning outcomes
You will (1) explain in plain terms how AI systems work and fail; (2) name the main attack types and the OWASP LLM Top 10 2025 risks; (3) use the NIST AI RMF functions to organise a decision; (4) compare adoption options including "not now"; and (5) record a decision with owners and review triggers.
1.1 How AI systems work, briefly
A machine-learning model learns patterns from examples and then makes predictions — ReceiptSort learns which receipts are travel, meals or supplies. A language model predicts text; an AI assistant wraps a language model with instructions (a system prompt), access to data (emails, documents) and tools (send email, create events). The assistant's behaviour depends on everything in its input, not only on your instructions.
For Tallis & Co this matters because InboxAide reads email written by people outside the firm. Any of that text becomes part of the assistant's input, alongside Omar's instructions and the firm's documents.
1.2 How they fail or are attacked
- Prompt injection. Text inside an email or document can instruct the assistant to do something else — for example, forward a client file. This was demonstrated against real applications (Greshake et al., arXiv:2302.12173).
- Data disclosure. The assistant may show information to people who should not see it.
- Excessive agency. An assistant with broad tools can cause broad harm when misled.
- Misinformation. Confident wrong answers.
- Evasion and poisoning. Classifiers can be fooled by crafted inputs or corrupted training data.
- Unbounded consumption. Abuse can run up costs.
The OWASP Top 10 for LLM Applications 2025 names these risks: LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure, LLM03 Supply Chain, LLM04 Data and Model Poisoning, LLM05 Improper Output Handling, LLM06 Excessive Agency, LLM07 System Prompt Leakage, LLM08 Vector and Embedding Weaknesses, LLM09 Misinformation, LLM10 Unbounded Consumption.
1.3 Organising the decision
NIST AI 100-1 (AI RMF) gives four functions. Govern: who owns the decision and the risk? Map: what is the system for, who is affected, what data and tools does it touch? Measure: what evidence do we have about the risks? Manage: what controls, monitoring and plans for when things go wrong? The Generative AI Profile (NIST AI 600-1) lists risks specific to generative systems to include in the Map step.
Omar used the four functions as headings for his first page of notes on InboxAide.
1.4 Options
Always compare more than "yes or no": (a) not now; (b) limited pilot — read and draft only, a few volunteers, no client data; (c) wider use with controls — sending requires a human click, shared documents limited; (d) full use. Prefer reversible options and expand with evidence.
For InboxAide, option (b) can be stopped at any time and teaches the firm something either way, while option (d) would expose shared client folders from the first day. For ReceiptSort the options are narrower, because it has no tools and reads no outside text. Write each option with its main risk and what it would cost to reverse.
1.5 Recording the decision
Write: the question, options, evidence, risks to clients and staff, chosen option, owner, conditions and review triggers (date, incident, vendor change).
Omar's record for InboxAide fits on one page: the question was whether to adopt the assistant, the chosen option was a four-week draft-only pilot, the owner is the managing partner, and the triggers are the end of the pilot, any incident or a vendor feature change. Keep the record where the owner and IT can both find it, and update it when a trigger fires rather than writing a new one.
1.6 Worked example (synthetic Tallis & Co case)
Omar maps InboxAide. It reads all email — including emails from outsiders, which can contain injected text — and can send email and access shared client folders. The worst plausible harm: client tax files sent to an attacker. He compares options and recommends (b): a four-week pilot with ten staff, draft-only (no sending), access to their own mailboxes only, no shared client folders. Owner: managing partner. Review triggers: end of pilot, any incident, or vendor feature change. ReceiptSort is lower risk (no tools, no outside text input to a language model) and goes to a normal software evaluation.
1.7 Errors and limits
- Deciding on vendor demos alone.
- Only "yes or no" options.
- No owner or review trigger.
Each error leads to a weak decision. A vendor demo shows the product at its best, with friendly inputs and no outside text, so it says little about prompt injection in real email. Only yes or no hides cheaper, reversible options such as Omar's draft-only pilot. Without an owner and review triggers, nobody notices when the vendor changes a feature or an incident occurs. Check your own decision record for all three.
Exercises
- EX001 — Plain terms. Explain in two sentences how an AI assistant differs from the model inside it. Rubric: 3 = instructions, data, tools around the model; 2 = two of these; 1 = vague; 0 = wrong.
- EX002 — Injection. Explain prompt injection to a partner in three sentences. Rubric: 3 = text in content steers assistant, example, consequence; 2 = two; 1 = one; 0 = wrong.
- EX003 — Disclosure. Give one InboxAide disclosure scenario. Rubric: 3 = specific data, viewer, path; 2 = two; 1 = vague; 0 = none.
- EX004 — Agency. Why does sending ability raise risk? Rubric: 3 = misled assistant acts externally, irreversible, scale; 2 = two; 1 = one; 0 = none.
- EX005 — OWASP. Map three InboxAide risks to OWASP entries. Rubric: 3 = three correct; 2 = two; 1 = one; 0 = none.
- EX006 — ReceiptSort. Name two threats to ReceiptSort. Rubric: 3 = evasion and poisoning with examples; 2 = names only; 1 = one; 0 = none.
- EX007 — Govern. Who should own the InboxAide decision and why? Rubric: 3 = accountable senior owner with authority over risk; 2 = owner only; 1 = IT by default; 0 = none.
- EX008 — Map. List four Map questions for InboxAide. Rubric: 3 = four apt; 2 = three; 1 = one or two; 0 = none.
- EX009 — Measure. What evidence would you want before wider use? Rubric: 3 = pilot incidents, injection tests, vendor documentation, user feedback; 2 = three; 1 = one; 0 = none.
- EX010 — Manage. List three controls for the pilot. Rubric: 3 = three apt; 2 = two; 1 = one; 0 = none.
- EX011 — Options. Write the four options with one benefit and one risk each. Rubric: 3 = four complete; 2 = three; 1 = one or two; 0 = none.
- EX012 — Reversibility. Why prefer reversible options? Rubric: 3 = learn safely, limited harm, cheap rollback; 2 = two; 1 = one; 0 = none.
- EX013 — Record. Write Omar's decision record. Rubric: 3 = question, options, evidence, choice, owner, triggers; 2 = four; 1 = two; 0 = none.
Synthesis
AI systems are models wrapped with instructions, data and tools; they can be steered, leak or overreach. Organise decisions with Govern, Map, Measure and Manage, compare reversible options and record owners and triggers.