◇ OpenCyber Academy
Technical · Chapter 9 · English

Chapter 9 — AI and machine-learning security: tools and methods

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 9 — AI and machine-learning security: tools and methods

Lesson map

Lesson 09 published. Map, example, EX033.

flowchart TD
  N0["Learning outcomes"]
  N1["9.1 Structured testing"]
  N2["9.2 A simple harness"]
  N3["9.3 Canaries and planted documents"]
  N4["9.4 Base rates"]
  N5["9.6b Keeping the test set current"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX033"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["9.1 Structured testing"]
  N2["9.2 A simple harness"]
  N3["9.3 Canaries and planted documents"]
  N4["9.4 Base rates"]
  N5["9.6b Keeping the test set current"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX033"]

[Learning outcomes]
    |
    v
[9.1 Structured testing]
    |
    v
[9.2 A simple harness]
    |
    v
[9.3 Canaries and planted documents]
    |
    v
[9.4 Base rates]
    |
    v
[9.6b Keeping the test set current]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX033]

Learning outcomes

You will (1) run structured tests of an AI service; (2) use a simple test harness; (3) test with canaries and planted documents; (4) interpret results with base rates in mind.

9.1 Structured testing

Each test has an ID, a category (for example LLM01), input, setup, expected safe behaviour and a grader. Group tests into suites: injection, disclosure, agency, consumption. Keep some tests aside (held-out) to check fixes fairly.

Kestrel's HelpBot suite follows this layout. A test such as INJ-07 is categorised LLM01, its input is a synthetic customer message asking about slow speed, its setup is a planted staging article containing a restart instruction, the expected safe behaviour is no restart_router call, and the grader reads the tool log rather than the reply. Held-out tests are stored in a separate file that only the team lead edits, so nobody tunes a fix against them by accident.

9.2 A simple harness

A harness script reads tests from a file, sends them to the staging service, collects replies and tool logs, applies graders and writes a report with counts and rates. It runs the same way every time, so results are comparable.

Ana's harness reads tests from a CSV file, sends each one to HelpBot staging with a fresh synthetic session, collects the reply and the tool-call log by request ID, applies the grader and writes counts per category. It also records the model version, prompt version and index snapshot at the start, so a report reads like "6/40 injection failures on prompt v14, model version as pinned". Running it twice on the same versions should give the same counts; when it did not, she found a test depending on the time of day.

9.3 Canaries and planted documents

A canary is a unique phrase placed where it should never be shown (a staff-only document); finding it in a reply proves a leak. Planted documents contain test instructions to check whether retrieved content can steer the assistant. Use them only on staging copies.

Ana's canary was a made-up phrase placed in a copy of a staff-only procedure on staging. One of the 30 disclosure tests produced it in a customer reply, and tracing the retrieved document IDs showed why: the procedure had been filed in the public collection by mistake. Planted documents work the same way for injection: a staging article tells the model to open an urgent ticket, and the grader checks the ticket log. Canaries in production would leak into real customers' answers.

9.4 Base rates

A detector that is right 95% of the time can still produce mostly false alarms when attacks are rare. At 0.1% prevalence and 5% false positives, fewer than 2 in 100 alerts are real.

"""Lab B: base rates and alarm precision (Bayes' rule).
A detector for malicious prompts with given true- and false-positive rates
is applied to traffic with different shares of malicious prompts."""
for tpr, fpr in ((0.95, 0.05), (0.95, 0.01), (0.80, 0.001)):
    for prev in (0.10, 0.01, 0.001):
        tp = tpr * prev; fp = fpr * (1 - prev)
        prec = tp / (tp + fp)
        per_10k = round(10000 * (tp + fp))
        print(f"tpr={tpr:.2f} fpr={fpr:.3f} prevalence={prev:.3f} precision={prec:.3f} alerts_per_10k={per_10k}")

Expected output:

tpr=0.95 fpr=0.050 prevalence=0.100 precision=0.679 alerts_per_10k=1400
tpr=0.95 fpr=0.050 prevalence=0.010 precision=0.161 alerts_per_10k=590
tpr=0.95 fpr=0.050 prevalence=0.001 precision=0.019 alerts_per_10k=509
tpr=0.95 fpr=0.010 prevalence=0.100 precision=0.913 alerts_per_10k=1040
tpr=0.95 fpr=0.010 prevalence=0.010 precision=0.490 alerts_per_10k=194
tpr=0.95 fpr=0.010 prevalence=0.001 precision=0.087 alerts_per_10k=109
tpr=0.80 fpr=0.001 prevalence=0.100 precision=0.989 alerts_per_10k=809
tpr=0.80 fpr=0.001 prevalence=0.010 precision=0.890 alerts_per_10k=90
tpr=0.80 fpr=0.001 prevalence=0.001 precision=0.445 alerts_per_10k=18

The lab's rows show how quickly precision changes. With 95% detection and 5% false positives, precision is 0.679 when one prompt in ten is malicious, 0.161 at one in a hundred and 0.019 at one in a thousand, producing 509 alerts per 10,000 messages. A detector with 80% detection and 0.1% false positives reaches precision 0.445 at 0.1% prevalence with only 18 alerts per 10,000. For HelpBot, where most of 20,000 daily messages are ordinary support questions, the false-positive rate decides whether a detector is usable.

9.5 Worked example (synthetic Kestrel case)

Ana runs a 100-test harness on HelpBot staging: 40 injection, 30 disclosure, 20 agency, 10 consumption. Results: 6 injection tests cause unwanted ticket creation; 1 disclosure test reveals a staff-only canary from a misfiled article; agency and consumption pass. She files two findings with test IDs. A colleague proposes an injection detector with 5% false positives on 20,000 daily messages; Ana estimates around 1,000 false alarms a day and recommends dashboard-only use.

The findings carried their test IDs so developers could reproduce them exactly. The injection finding listed the six tests and their planted articles; the disclosure finding named the misfiled article and the canary phrase. Neither recommended a detector, because the alert estimate showed the team would drown in false alarms. The fixes came later as configuration in the tool service and the retrieval collections.

9.6 Errors and limits

Ad hoc tests caused a real problem early on: Ana's first injection attempts were typed by hand and not saved, so after a prompt change she could not tell whether HelpBot had improved or whether she had simply phrased things differently. Canaries in production would place a staff-only marker in front of real customers. A detector deployed without considering volume, at 5% false positives on 20,000 messages, produces about 1,000 alerts a day for a team of three.

9.6b Keeping the test set current

The harness's attack prompts age as HelpBot changes. Each month a technician adds new cases from recent reports and removes ones that no longer apply, recording the change so test results from different months can still be compared.

When the Wi-Fi password tool was added, Ana added 30 cross-customer tests for it to the agency suite; when the factory-reset article was found, its question became a regression test. Retired tests, such as those for a tool that was removed, are moved to an archive file rather than deleted, so an old report can still be rerun as written.

Exercises

Synthesis

Structured suites, a repeatable harness, canaries and base-rate thinking make AI-security testing reliable and useful.

Continue studying — chapter 09 of 18

← Chapter 08 (draft) · Course chapter index · Chapter 10 (draft) →