## Chapter 1 — AI and machine-learning security: foundations and decisions ### Learning outcomes You will (1) explain in plain terms how AI systems work and fail; (2) name the main attack types and the OWASP LLM Top 10 2025 risks; (3) use the NIST AI RMF functions to organise a decision; (4) compare adoption options including "not now"; and (5) record a decision with owners and review triggers. ### 1.1 How AI systems work, briefly A machine-learning model learns patterns from examples and then makes predictions — ReceiptSort learns which receipts are travel, meals or supplies. A language model predicts text; an AI assistant wraps a language model with instructions (a system prompt), access to data (emails, documents) and tools (send email, create events). The assistant's behaviour depends on everything in its input, not only on your instructions. For Tallis & Co this matters because InboxAide reads email written by people outside the firm. Any of that text becomes part of the assistant's input, alongside Omar's instructions and the firm's documents. ### 1.2 How they fail or are attacked - **Prompt injection.** Text inside an email or document can instruct the assistant to do something else — for example, forward a client file. This was demonstrated against real applications (Greshake et al., arXiv:2302.12173). - **Data disclosure.** The assistant may show information to people who should not see it. - **Excessive agency.** An assistant with broad tools can cause broad harm when misled. - **Misinformation.** Confident wrong answers. - **Evasion and poisoning.** Classifiers can be fooled by crafted inputs or corrupted training data. - **Unbounded consumption.** Abuse can run up costs. The OWASP Top 10 for LLM Applications 2025 names these risks: LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure, LLM03 Supply Chain, LLM04 Data and Model Poisoning, LLM05 Improper Output Handling, LLM06 Excessive Agency, LLM07 System Prompt Leakage, LLM08 Vector and Embedding Weaknesses, LLM09 Misinformation, LLM10 Unbounded Consumption. ### 1.3 Organising the decision NIST AI 100-1 (AI RMF) gives four functions. **Govern**: who owns the decision and the risk? **Map**: what is the system for, who is affected, what data and tools does it touch? **Measure**: what evidence do we have about the risks? **Manage**: what controls, monitoring and plans for when things go wrong? The Generative AI Profile (NIST AI 600-1) lists risks specific to generative systems to include in the Map step. Omar used the four functions as headings for his first page of notes on InboxAide. ### 1.4 Options Always compare more than "yes or no": (a) not now; (b) limited pilot — read and draft only, a few volunteers, no client data; (c) wider use with controls — sending requires a human click, shared documents limited; (d) full use. Prefer reversible options and expand with evidence. For InboxAide, option (b) can be stopped at any time and teaches the firm something either way, while option (d) would expose shared client folders from the first day. For ReceiptSort the options are narrower, because it has no tools and reads no outside text. Write each option with its main risk and what it would cost to reverse. ### 1.5 Recording the decision Write: the question, options, evidence, risks to clients and staff, chosen option, owner, conditions and review triggers (date, incident, vendor change). One page is enough for that record. Omar's record for InboxAide fits on one page: the question was whether to adopt the assistant, the chosen option was a four-week draft-only pilot, the owner is the managing partner, and the triggers are the end of the pilot, any incident or a vendor feature change. Keep the record where the owner and IT can both find it, and update it when a trigger fires rather than writing a new one. ### 1.6 Worked example (synthetic Tallis & Co case) Omar maps InboxAide. It reads all email — including emails from outsiders, which can contain injected text — and can send email and access shared client folders. The worst plausible harm: client tax files sent to an attacker. He compares options and recommends (b): a four-week pilot with ten staff, draft-only (no sending), access to their own mailboxes only, no shared client folders. Owner: managing partner. Review triggers: end of pilot, any incident, or vendor feature change. ReceiptSort is lower risk (no tools, no outside text input to a language model) and goes to a normal software evaluation. ### 1.7 Errors and limits - Deciding on vendor demos alone. - Only "yes or no" options. - No owner or review trigger. Each error leads to a weak decision. A vendor demo shows the product at its best, with friendly inputs and no outside text, so it says little about prompt injection in real email. Only yes or no hides cheaper, reversible options such as Omar's draft-only pilot. Without an owner and review triggers, nobody notices when the vendor changes a feature or an incident occurs. Check your own decision record for all three. ### Exercises - **EX001 — Plain terms.** Explain in two sentences how an AI assistant differs from the model inside it. Rubric: 3 = instructions, data, tools around the model; 2 = two of these; 1 = vague; 0 = wrong. - **EX002 — Injection.** Explain prompt injection to a partner in three sentences. Rubric: 3 = text in content steers assistant, example, consequence; 2 = two; 1 = one; 0 = wrong. - **EX003 — Disclosure.** Give one InboxAide disclosure scenario. Rubric: 3 = specific data, viewer, path; 2 = two; 1 = vague; 0 = none. - **EX004 — Agency.** Why does sending ability raise risk? Rubric: 3 = misled assistant acts externally, irreversible, scale; 2 = two; 1 = one; 0 = none. - **EX005 — OWASP.** Map three InboxAide risks to OWASP entries. Rubric: 3 = three correct; 2 = two; 1 = one; 0 = none. - **EX006 — ReceiptSort.** Name two threats to ReceiptSort. Rubric: 3 = evasion and poisoning with examples; 2 = names only; 1 = one; 0 = none. - **EX007 — Govern.** Who should own the InboxAide decision and why? Rubric: 3 = accountable senior owner with authority over risk; 2 = owner only; 1 = IT by default; 0 = none. - **EX008 — Map.** List four Map questions for InboxAide. Rubric: 3 = four apt; 2 = three; 1 = one or two; 0 = none. - **EX009 — Measure.** What evidence would you want before wider use? Rubric: 3 = pilot incidents, injection tests, vendor documentation, user feedback; 2 = three; 1 = one; 0 = none. - **EX010 — Manage.** List three controls for the pilot. Rubric: 3 = three apt; 2 = two; 1 = one; 0 = none. - **EX011 — Options.** Write the four options with one benefit and one risk each. Rubric: 3 = four complete; 2 = three; 1 = one or two; 0 = none. - **EX012 — Reversibility.** Why prefer reversible options? Rubric: 3 = learn safely, limited harm, cheap rollback; 2 = two; 1 = one; 0 = none. - **EX013 — Record.** Write Omar's decision record. Rubric: 3 = question, options, evidence, choice, owner, triggers; 2 = four; 1 = two; 0 = none. ### Synthesis AI systems are models wrapped with instructions, data and tools; they can be steered, leak or overreach. Organise decisions with Govern, Map, Measure and Manage, compare reversible options and record owners and triggers. ### Continue studying — lesson 01 (chapter 01 of 20) [← Course home](../../README.md) · [Course chapter index](../SUBJECTS.md) · [Chapter 02 →](#ch-02) Continue to the next authored lesson. ## Worked answers **EX001.** The model only predicts text. The assistant wraps it with instructions, access to data such as email and documents, and tools such as sending and calendar, so it can read and act. **EX002.** "Anyone who emails us can hide instructions in the message, and the assistant may follow them as if they came from us. For example, an email could tell it to attach a client's tax return to a reply. If the assistant can send email, that could leak client data without anyone noticing." **EX003.** A junior employee asks InboxAide to summarise "recent client issues". Because it was connected to the shared partner folder, the summary includes a confidential fee dispute the junior should not see. **EX004.** A misled assistant can act outside the firm, and sent email cannot be recalled. It can also act quickly and many times, so harm scales. **EX005.** Hidden instructions in client email: LLM01. Revealing data across users: LLM02. Sending and sharing ability: LLM06. **EX006.** - Evasion: receipts altered so personal meals look like supplies. - Poisoning: mislabelled training receipts teach wrong categories. **EX007.** The managing partner (or a designated partner). They are accountable for client confidentiality and can accept or refuse the risk. IT advises but should not own a business risk alone. **EX008.** 1. What is InboxAide for? 2. Whose data does it read? 3. Which actions can it take? 4. Who is affected if it errs, including clients and staff? **EX009.** Pilot incident and near-miss reports, injection test results with rates, vendor documentation on data use and controls, and user feedback on usefulness and errors. **EX010.** Draft-only mode (sending disabled); access limited to each user's own mailbox; action logs reviewed weekly. **EX011.** | Option | Benefit | Risk | |---|---|---| | (a) Not now | No new risk | Staff may use unapproved tools | | (b) Pilot | Learn safely | Limited value | | (c) Wider use with controls | Most value at bounded risk | Approval fatigue | | (d) Full use | Maximum convenience | Large harm if steered | **EX012.** They let you learn from real use while harm stays limited. If problems appear, rollback is cheap, and each expansion can be based on evidence. **EX013.** - Question: adopt InboxAide? - Options: a–d. - Evidence: vendor documentation, the flow diagram, and pending tests. - Chosen: (b), a four-week pilot with ten staff, draft-only, own mailboxes, no shared client folders. - Owner: managing partner. - Triggers: pilot end, any incident, or a vendor feature change.