◇ OpenCyber Academy
Bachelor's · Chapter 15 · English

Chapter 15 — Evidence and measurement

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 15 — Evidence and measurement

Lesson map

Lesson 15 published. Map, example, EX043.

flowchart TD
  N0["Learning outcomes"]
  N1["15.1 Metrics that match questions"]
  N2["15.2 Test sets and leakage"]
  N3["15.3 Evidence handling"]
  N4["15.4 Reporting"]
  N5["15.4b Baselines and comparisons"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX043"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["15.1 Metrics that match questions"]
  N2["15.2 Test sets and leakage"]
  N3["15.3 Evidence handling"]
  N4["15.4 Reporting"]
  N5["15.4b Baselines and comparisons"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX043"]

[Learning outcomes]
    |
    v
[15.1 Metrics that match questions]
    |
    v
[15.2 Test sets and leakage]
    |
    v
[15.3 Evidence handling]
    |
    v
[15.4 Reporting]
    |
    v
[15.4b Baselines and comparisons]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX043]

Learning outcomes

You will (1) choose metrics that match security questions; (2) design test sets that avoid leakage; (3) collect and preserve evidence properly; and (4) write measurement results with conditions and uncertainty.

15.1 Metrics that match questions

"Does the chatbot leak other patients' data?" needs a leakage rate on a test suite of attempts. "Does the filter catch injections?" needs TPR at a fixed FPR. "Does the defence harm usefulness?" needs task success on normal requests. Use several metrics; one number rarely answers a security question.

Fixing the false-positive rate before comparing filters ensures that a filter cannot look better simply by flagging more conversations. Staff time spent reviewing false alarms is a real cost for the clinics.

15.2 Test sets and leakage

Keep test data separate from anything used to build or tune the system. Near-duplicates across sets inflate results (Kapoor and Narayanan, arXiv:2207.07048). For prompt attacks, build test cases from templates you did not use while developing defences.

If Rina tuned a defence on the same injection templates she later tested it on, her success rate would measure memorisation of those templates rather than the defence's general strength. She keeps a separate set of attack templates written by a classmate and opens it only for the final evaluation. Checking for near-duplicates between sets, such as the same attack with a changed name, is part of building them.

15.3 Evidence handling

Record inputs, outputs, versions (model, prompt, code) and timestamps. Store evidence with integrity hashes and access control. In incident work, preserve original logs before analysis.

Versions matter because the same prompt can give different results after a model update by the provider. Rina records the model identifier returned by the API, the system prompt's hash and the code commit for each test run. Hashing the transcripts lets her show later that they were not edited. Original logs are copied and hashed before any analysis, and analysis is done on the copy.

15.4 Reporting

State the question, method, sample, results with intervals, conditions and limitations. Arp et al. (arXiv:2010.09470) list pitfalls such as sampling bias and inappropriate baselines; check yours against them.

Rina's leakage report follows this order: question, 400 synthetic attempts, results for two versions with Wilson intervals, conditions such as English-only prompts, and limitations. Checking against Arp et al.'s list, she found one pitfall in her draft: she had compared version 2 against version 1 tested months earlier with a different model release, which she then repeated on the same release.

15.4b Baselines and comparisons

A number means little without a baseline. Report the undefended system's result next to the defended one, and a simple strong alternative (for example, a tool allowlist) next to your elaborate defence. If the simple alternative achieves most of the benefit, say so: it is useful knowledge for practitioners and protects you from overclaiming.

For Ask Lumen, a strong simple baseline is removing the cancel tool entirely, which no injection can then misuse. Any proposed defence should be compared with that option.

15.5 Worked example (synthetic LCH case)

Rina measured Ask Lumen's leakage with 400 synthetic attempts to obtain another patient's appointment. Version 1 leaked in 14 cases (3.5%, Wilson 95% interval 2.1–5.8%). After the delegated-token fix, version 2 leaked in 0 of 400 (upper bound about 0.95%). She stored transcripts with hashes and noted that attempts were in English only.

The interval matters: 14 of 400 is consistent with a true rate between about 2.1% and 5.8%, so the fix's effect is clear even allowing for chance. Zero leaks in 400 does not mean zero risk; the upper bound of about 0.95% is what the test can support. Noting the English-only limit tells readers that other languages remain untested.

15.6 Errors and limits

Tuning on the test set inflates results without improving the system. Reporting without versions makes results impossible to reproduce once the model or prompt changes. Ignoring language or format limits leads readers to assume coverage that was never tested, for example attacks written in another language or hidden in a document format the test suite did not include, such as a scanned PDF.

Exercises

Synthesis

Good measurement asks the right question, uses clean test data, preserves evidence and reports results with conditions and uncertainty.

Continue studying — chapter 15 of 40

← Chapter 14 (draft) · Course chapter index · Chapter 16 (draft) →