Chapter 42 — AI and machine-learning security: applied research
Lesson map
Lesson 42 published. Map, example, EX124.
flowchart TD
N0["Learning outcomes"]
N1["42.1 From problem to question"]
N2["42.2 Efficient literature review"]
N3["42.3 Valid small experiments"]
N4["42.4 Reporting for practitioners"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX124"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["42.1 From problem to question"]
N2["42.2 Efficient literature review"]
N3["42.3 Valid small experiments"]
N4["42.4 Reporting for practitioners"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX124"]
[Learning outcomes]
|
v
[42.1 From problem to question]
|
v
[42.2 Efficient literature review]
|
v
[42.3 Valid small experiments]
|
v
[42.4 Reporting for practitioners]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX124]
Learning outcomes
You will (1) frame an applied research question from an engineering problem; (2) review literature efficiently using shared taxonomies; (3) design a small, valid experiment; and (4) report findings with limitations useful to practitioners.
42.1 From problem to question
Engineering problems become research questions when the answer is not known and would generalise: "Does marking untrusted content with provenance labels reduce injection proposals in our agent, and by how much compared with a dual-model pattern?" Keep scope to what one semester can answer.
A good question names the comparison, the measure and the setting, so it can be answered with a planned experiment. 'Are labels useful?' cannot be answered; 'how many injection proposals do labels prevent in Wrench compared with a dual-model design, on held-out bulletins' can. Keeping scope small leaves time to do the experiment properly, including the adaptive attack that many projects skip.
42.2 Efficient literature review
Start from authoritative overviews — NIST AI 100-2 E2025 for adversarial ML, OWASP LLM 2025 for application risks, MITRE ATLAS for techniques — then read key papers: Greshake et al. (arXiv:2302.12173) and Liu et al. (arXiv:2306.05499) on prompt injection; Chen et al. (arXiv:2402.06363) on structured-query defences; Carlini et al. (arXiv:1902.06705) on evaluation. Record threat model, method, metric and limits for each.
Overviews map the field quickly and point to the papers worth reading closely. For each paper, a short note records the threat model, the method, the metric and the limits, so papers can be compared on the same terms. Sam stops reading when new papers no longer change the plan.
42.3 Valid small experiments
Pre-specify hypotheses and metrics; use held-out test sets; compare against strong baselines; run several seeds; report paired differences with intervals; include an adaptive attack that targets your method. Check common pitfalls in Arp et al. (arXiv:2010.09470).
A small experiment can still be valid if it is honest about its limits. Pre-specifying prevents choosing the metric that flatters the method. Strong baselines stop a method from looking good against a weak comparison. Several seeds show whether a difference is stable. An adaptive attack, written by someone who knows how the method works, is the most important test, because attackers will know too.
42.4 Reporting for practitioners
State the setting precisely, the effect with its interval, the cost, the conditions under which it failed, and what a practitioner should do with the result.
Practitioners want to know whether to adopt a method, under what conditions and at what cost. A report that ends with a p-value leaves them to work this out. Stating where the method failed is particularly useful, since it tells readers when not to rely on it. Sam ends each applied report with a short recommendation and its conditions, written for an engineer who will not read the methods.
42.5 Worked example (synthetic NRS case)
Sam's applied question compared provenance labelling alone, dual-model, and both, on 400 synthetic injected bulletins and 400 benign tasks, five seeds each. Labelling reduced injection proposals from 14% to 9% (paired CI 3–7 points); dual-model to 2%; both to 1.5%. An adaptive payload that imitated the label format raised labelling-only proposals back to 13%, while dual-model held at 2.5%. Conclusion for practitioners: labels help a little and are easily imitated; constrained interfaces help more; neither replaces enforcement.
The adaptive attack changed the conclusion: without it, labelling would have looked like a solid defence.
42.6 Errors and limits
- Questions too broad for the time available.
- No adaptive attack against the proposed method.
- Reports without practitioner guidance.
Questions that are too broad end with vague results or unfinished experiments. A method tested only against attacks written before it existed will look stronger than it is, as the label-imitating payload showed. Reports that stop at statistics leave practitioners unsure what to do. Sam checks proposals for a single answerable question, an adaptive attack in the plan and a practitioner summary in the outline before any experiment runs.
Exercises
- EX124 — Question. Turn "our classifier misses some injections" into an applied research question. Rubric: 3 = specific, comparative, measurable, feasible; 2 = three; 1 = vague; 0 = none.
- EX125 — Adaptive attack. Why did the label-imitating payload matter in Sam's study? Rubric: 3 = shows labelling is weak against an attacker who knows it; changes conclusion; 2 = partial; 1 = vague; 0 = wrong.
- EX126 — Practitioner sentence. Write one practitioner recommendation from Sam's results. Rubric: 3 = actionable, conditional, with evidence; 2 = partial; 1 = overclaims; 0 = none.
Synthesis
Applied research answers engineering questions with small, valid experiments — including attacks on your own idea — and reports results practitioners can use.
Continue studying — chapter 42 of 50
← Chapter 41 (draft) · Course chapter index · Chapter 43 (draft) →