Chapter 10 — Research reading and critique | Postgraduate AI and machine-learning security

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 10 published. Map, example, EX046.

flowchart TD
  N0["Learning outcomes"]
  N1["10.1 Reading efficiently"]
  N2["10.2 Critique questions"]
  N3["10.3 Transfer to practice"]
  N4["10.4 Keeping up"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX046"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["10.1 Reading efficiently"]
  N2["10.2 Critique questions"]
  N3["10.3 Transfer to practice"]
  N4["10.4 Keeping up"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX046"]

[Learning outcomes]
    |
    v
[10.1 Reading efficiently]
    |
    v
[10.2 Critique questions]
    |
    v
[10.3 Transfer to practice]
    |
    v
[10.4 Keeping up]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX046]

Learning outcomes

You will (1) read AI-security papers efficiently; (2) critique threat models, methods and evaluations; (3) judge whether results transfer to your setting; (4) track a fast-moving literature sustainably; and (5) write a practitioner's critique.

10.1 Reading efficiently

Read in three passes: abstract, figures and conclusion to decide relevance; threat model and method to understand the claim; evaluation and limitations to judge it. Keep a one-paragraph note per paper: claim, threat model, evidence, limits, relevance to your systems.

Amira's critique of the preprint in the worked example below used the second and third passes to find the missing adaptive attacker. File each note where the team can find it. Keep the notes short enough to read in a minute, and tag each with the systems it affects, so they can be found when a decision needs them.

10.2 Critique questions

What can the attacker do and know? Is the threat model realistic for your deployment? Are baselines and adaptive attacks included? Are results measured by effects and reported with uncertainty? Would a small change in setting break the result? Did the authors release code and data? Amira's critique of the preprint in 10.5 found three of these missing: an adaptive attacker, success measured by tool calls rather than the model's text, and released code. Papers such as Carlini et al. (arXiv:1902.06705) give a checklist for robustness evaluations; Arp et al. (arXiv:2010.09470) list pitfalls in machine learning for security, including sampling bias, data snooping and inappropriate baselines.

10.3 Transfer to practice

An attack demonstrated on an open model with white-box access may not apply to an API-only deployment — but transfer results such as Zou et al. (arXiv:2307.15043) show that some attacks found on open models work on closed ones. A defence evaluated on a public benchmark may fail on your traffic. State transfer assumptions explicitly.

For Corvane, a defence tested only on public benchmarks would need testing on carrier emails before being trusted. Write each transfer assumption in the note. Revisit it after testing.

10.4 Keeping up

Follow a small set of sources: NIST and OWASP updates, MITRE ATLAS case studies, major security and ML conferences, and preprint servers with skepticism. Hold a monthly reading session where one person presents a paper's relevance to company systems.

At Corvane, the monthly session keeps the reading routine small enough to sustain. Rotate the presenter, and record in each note what changed, if anything, in Corvane's systems or plans as a result. Drop sources that rarely bring anything relevant, and add one when a colleague finds a better one.

10.5 Worked example (synthetic Corvane case)

A colleague circulates a preprint claiming a new filter "eliminates prompt injection". Amira's critique: the evaluation uses 200 public injection strings with no adaptive attacker; success is measured by the model's text rather than tool calls; no false-refusal rate is reported; code is not released. Her conclusion: interesting as a monitoring signal; not evidence to remove the policy service. She contrasts it with Greshake et al. (arXiv:2302.12173), whose demonstrations against real applications motivated Corvane's design.

The critique decided what the preprint could and could not justify at Corvane.

10.6 Errors and limits

Citing abstracts without reading evaluations.
Assuming results transfer without checking threat models.
Chasing every preprint.

Each error lets weak research steer decisions. An abstract can claim more than the evaluation shows, as the preprint's "eliminates" did. A result on an open model with white-box access may not apply to an API-only deployment. Chasing every preprint uses time better spent on a few relevant papers each month. Read your notes against these three before you cite a paper in a decision record.

Exercises

EX046 — Note. Write a five-field note on Shokri et al. (arXiv:1610.05820). Rubric: 3 = claim, threat model, evidence, limits, relevance; 2 = four; 1 = two; 0 = none.
EX047 — Critique. Apply four critique questions to the preprint in §10.5. Rubric: 3 = four answered with evidence; 2 = three; 1 = one or two; 0 = none.
EX048 — Transfer. When does a white-box attack matter for an API-only system? Rubric: 3 = transfer, surrogate models, open-weight base; 2 = one or two; 1 = vague; 0 = never.
EX049 — Pitfalls. Name three pitfalls from Arp et al. and an example of each in AI-security evaluation. Rubric: 3 = three with examples; 2 = two; 1 = one; 0 = none.
EX050 — Reading plan. Design a sustainable reading routine for a five-person team. Rubric: 3 = sources, cadence, roles, output; 2 = three; 1 = vague; 0 = none.

Synthesis

Practitioners read research for its threat model and evidence, not its headline. Critique systematically, state transfer assumptions and keep a sustainable routine.

Continue studying — chapter 10 of 12

← Chapter 09 (draft) · Course chapter index · Chapter 11 (draft) →