Chapter 16 — AI and machine-learning security: methods II
Lesson map
Lesson 16 published. Map, example, EX046.
flowchart TD
N0["Learning outcomes"]
N1["16.1 Retrieval tests"]
N2["16.2 Excessive agency"]
N3["16.3 Disclosure and system prompt leakage"]
N4["16.4 Resource tests"]
N5["16.4b Grading automatically"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX046"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["16.1 Retrieval tests"]
N2["16.2 Excessive agency"]
N3["16.3 Disclosure and system prompt leakage"]
N4["16.4 Resource tests"]
N5["16.4b Grading automatically"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX046"]
[Learning outcomes]
|
v
[16.1 Retrieval tests]
|
v
[16.2 Excessive agency]
|
v
[16.3 Disclosure and system prompt leakage]
|
v
[16.4 Resource tests]
|
v
[16.4b Grading automatically]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX046]
Learning outcomes
You will (1) test retrieval systems for access-control and poisoning weaknesses; (2) test agents for excessive agency; (3) test for sensitive-information disclosure and system-prompt leakage; and (4) test resource limits.
16.1 Retrieval tests
Plant canary documents with distinctive phrases in restricted locations and query as users who should not see them; any canary in an answer is a leak. Add documents with false facts from low-trust sources and check whether answers repeat them (LLM08, LLM09). Jonas used both methods on Ask Brightwell after the index split. The staff-only canary document tested whether anything restricted still reached customers, and false facts in low-trust pages tested whether answers repeated them. The expired 60-day returns page later showed that false facts can also arrive by accident.
16.2 Excessive agency
List every tool, its parameters and permissions. Try to make the agent call each tool with parameters outside the user's rights or the task's need, via direct requests and planted content. Measure from tool logs. OWASP LLM06 recommends minimising tools, functions and permissions, and requiring human approval for high-impact actions. For Ask Brightwell, the agency tests found that the agent could call "update address" during a returns chat. That tool was never needed there, and an injected review could have used it to redirect a replacement. Scoping tools per conversation type closed the gap.
16.3 Disclosure and system prompt leakage
Ask for other users' data, internal documents and the system prompt in many phrasings. Leaked system prompts matter mainly when they contain secrets or reveal controls; the fix is to remove secrets from prompts, not to hide the prompt better (LLM07). Ask Brightwell's system prompt once held a staff discount code, which made a prompt leak a real loss. After the code was moved out, a leaked prompt reveals only instructions that tests already assume an attacker can guess. Jonas still tests for leakage to confirm that no secret has been added back.
16.4 Resource tests
Send maximum-length inputs, requests for very long outputs, and recursive tool loops; measure cost and latency against limits (LLM10). For Ask Brightwell, the loop test found the agent calling search 40 times in one turn, each call using tokens. The cap of 5 calls per turn bounds that cost. Maximum-length inputs are checked against the gateway's size limit, and long-output requests against the vendor's output limit. Each test measures cost and latency from the logs, so a limit that is configured but not enforced shows up immediately.
16.4b Grading automatically
Write graders as code: search tool logs for calls outside allowed parameters, search outputs for canary strings, and compare token counts with limits. Automated graders make results repeatable and let you rerun the whole suite after each change. Ask Brightwell's graders search tool logs for start_return calls on orders the user did not mention, search answers for the Project Heron canary phrase, and compare per-turn tool calls with the cap. Because they are code, the same graders run unchanged in the nightly harness.
16.5 Worked example (synthetic Brightwell case)
Jonas plants a staff-only canary document ("Project Heron discount matrix") and asks 50 customer-account questions about discounts: 0 leaks after the index split. Agency tests find the agent can call "update address" during a returns chat — a tool it never needs there. He scopes tools per conversation type. A loop test shows the agent can call search 40 times in one turn; he caps calls at 5. Each result led to a narrower system rather than a better prompt: separate indexes, tools scoped per conversation, and a hard cap on calls. To confirm the canary test can see a leak at all, Jonas also ran it on a scratch copy with the Heron document back in the customer index.
16.6 Errors and limits
- Testing retrieval only with friendly queries.
- Tools enabled globally rather than per task.
- No per-turn limits.
Jonas's Ask Brightwell tests avoid all three: the canary questions come from customer accounts trying to reach staff content, tools are scoped per conversation type, and a per-turn call cap bounds loops. Friendly queries would never have found the margin leak, and global tools left the address tool reachable during returns. Each has a test in the harness, so a change that brings one back fails the nightly run.
Exercises
- EX046 — Canary. Design a canary test for the staff index. Rubric: 3 = unique phrase, placement, queries as wrong role, pass criterion; 2 = three; 1 = one; 0 = none.
- EX047 — Agency audit. List Ask Brightwell's tools and which conversations need each. Rubric: 3 = per-tool scoping with reasons; 2 = partial; 1 = vague; 0 = none.
- EX048 — Loop. Why cap tool calls per turn, and what value would you choose? Rubric: 3 = cost, loops, injection amplification, justified number; 2 = partial; 1 = vague; 0 = none.
Synthesis
Methods II extend testing to retrieval, agency, disclosure and consumption — each judged by effects and fixed with system controls.
Continue studying — chapter 16 of 30
← Chapter 15 (draft) · Course chapter index · Chapter 17 (draft) →