Chapter 33 — Research methods II
Lesson map
Lesson 33 published. Map, example, EX097.
flowchart TD
N0["Learning outcomes"]
N1["33.1 Experimental design"]
N2["33.2 Analysis"]
N3["33.3 Pitfalls"]
N4["33.4 Threats to validity"]
N5["33.4b Adaptive evaluation"]
N6["33.6b Preregistering the comparison"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX097"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["33.1 Experimental design"]
N2["33.2 Analysis"]
N3["33.3 Pitfalls"]
N4["33.4 Threats to validity"]
N5["33.4b Adaptive evaluation"]
N6["33.6b Preregistering the comparison"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX097"]
[Learning outcomes]
|
v
[33.1 Experimental design]
|
v
[33.2 Analysis]
|
v
[33.3 Pitfalls]
|
v
[33.4 Threats to validity]
|
v
[33.4b Adaptive evaluation]
|
v
[33.6b Preregistering the comparison]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX097]
Learning outcomes
You will (1) design controlled experiments with multiple conditions and seeds; (2) analyse paired comparisons with bootstrap intervals; (3) handle multiple comparisons; and (4) recognise and avoid common pitfalls in ML security research.
33.1 Experimental design
Define conditions (baseline, defence A, defence B, A + B), hold everything else fixed, randomise task order and use several seeds. Pre-register hypotheses and metrics in a dated document before running.
Holding everything else fixed means the same model version, the same document set and the same prompts across all four Ask Lumen conditions. Randomising order prevents later tasks from benefiting from caching or rate-limit effects. Rina's dated plan was stored in the course repository before her first run, so the timestamp shows it came first.
33.2 Analysis
Use paired designs: the same tasks under every condition. Bootstrap the difference between conditions to get intervals. When comparing many conditions, control false discoveries (for example with Holm–Bonferroni) or label additional comparisons as exploratory.
"""Lab S: reporting attack-success rates honestly: Wilson intervals,
seed variation and a paired bootstrap. Synthetic trial outcomes only."""
import math, random
def wilson(k, n, z=1.96):
p = k/n; d = 1 + z*z/n
c = (p + z*z/(2*n))/d; h = z*math.sqrt(p*(1-p)/n + z*z/(4*n*n))/d
return c - h, c + h
for k, n in ((3, 20), (30, 200), (300, 2000)):
lo, hi = wilson(k, n); print(f"successes={k}/{n} rate={k/n:.3f} wilson95=[{lo:.3f}, {hi:.3f}]")
# five seeds of a synthetic defence evaluation: per-seed attack success rate
rates = []
for seed in range(5):
r = random.Random(seed); rates.append(sum(r.random() < 0.12 for _ in range(200))/200)
m = sum(rates)/5; sd = math.sqrt(sum((x-m)**2 for x in rates)/4)
print("per-seed rates:", " ".join(f"{x:.3f}" for x in rates), f"mean={m:.3f} sd={sd:.3f}")
# paired bootstrap: baseline vs defence on the same 300 synthetic prompts
r = random.Random(42)
base = [r.random() < 0.30 for _ in range(300)]
defn = [b and r.random() < 0.6 for b in base] # defence blocks ~40% of successes
diffs = []
for _ in range(2000):
idx = [r.randrange(300) for _ in range(300)]
diffs.append(sum(base[i] for i in idx)/300 - sum(defn[i] for i in idx)/300)
diffs.sort()
print(f"baseline_asr={sum(base)/300:.3f} defended_asr={sum(defn)/300:.3f} "
f"reduction_ci95=[{diffs[50]:.3f}, {diffs[1949]:.3f}]")
Expected output:
successes=3/20 rate=0.150 wilson95=[0.052, 0.360]
successes=30/200 rate=0.150 wilson95=[0.107, 0.206]
successes=300/2000 rate=0.150 wilson95=[0.135, 0.166]
per-seed rates: 0.110 0.130 0.135 0.105 0.145 mean=0.125 sd=0.017
baseline_asr=0.350 defended_asr=0.190 reduction_ci95=[0.120, 0.203]
In the lab, the reduction from 0.350 to 0.190 has a bootstrap interval of about 0.120 to 0.203, so the defence clearly helps. The per-seed rates, from 0.105 to 0.145, show why several seeds are needed. A single seed could suggest a larger or smaller effect than the average shows, and a reader would have no way of knowing which. For Ask Lumen, Rina used five seeds per configuration.
33.3 Pitfalls
Arp et al. (arXiv:2010.09470) list pitfalls including sampling bias, label inaccuracy, data snooping, spurious correlations, biased parameter selection, inappropriate baselines, base-rate fallacy, lab-only evaluation, inappropriate threat models and inappropriate performance measures. Use the list as a checklist before and after experiments.
Applied to Rina's four-configuration study, the base-rate fallacy was relevant to the detector discussion and lab-only evaluation to every result, since staging traffic is synthetic. Data snooping was avoided because the held-out injection set was opened only once. She recorded which pitfalls applied and what she did about each.
33.4 Threats to validity
Internal (something else caused the effect), construct (the metric does not measure the property), external (results do not transfer) and conclusion (statistics inadequate). Write them down with what you did about each.
For Rina's study, external validity was the main concern: all injected documents were English, and real partner leaflets include other languages. Construct validity was checked by confirming that her false-statement grader matched human labels. Internal validity was supported by holding the model version fixed throughout, so a provider update could not explain the differences.
33.4b Adaptive evaluation
For any defence you propose, include at least one attack designed with knowledge of it. Without that, results describe resistance to yesterday's attacks only (Carlini et al., arXiv:1902.06705).
Rina's adaptive set for the grounding check used documents with plausible but false times that matched the structured file's format. A check that only compared formats would accept them; hers compared values, so the false times were caught. She also tried documents stating a clinic's hours had changed "from next week", which the check could not verify. Those cases went to human review, which she recorded as a known limit of the grounding approach.
33.5 Worked example (synthetic LCH case)
Rina compared four Ask Lumen configurations on 500 injected and 500 benign tasks with five seeds. Grounding check alone reduced false clinic statements from 9.0% to 3.2%; source labelling alone to 7.1%; both to 2.4%. The paired bootstrap interval for "both versus grounding alone" was 0.1–1.5 points — a small but positive improvement — and she noted that injected documents were all English.
The interval excludes zero, so the combination helps, but the gain is small enough that LCH weighed it against source labelling's maintenance cost. The English-only limit was stated in the summary.
33.6 Errors and limits
- No pre-registration.
- Many comparisons reported as if each were the only one.
- Ignoring the pitfall checklist.
Without a dated plan, nothing shows the hypothesis came before the data, and readers must trust the author. Reporting six pairwise comparisons as if each were the only test raises the chance that one looks significant by luck; Rina used Holm–Bonferroni for her three planned comparisons. Skipping the pitfall list lets familiar mistakes reach the final report unnoticed, where they are costly to correct.
33.6b Preregistering the comparison
Before running the prompt-injection study, the LCH team writes down the hypothesis, the attack set, the metric and the threshold that would count as improvement, and stores that plan with a date. Results are then compared with the plan, which stops the team from choosing a flattering metric after seeing the data.
Rina's plan named the false-statement rate as the primary metric and completed bookings as secondary. It also fixed the number of seeds and the threshold for improvement, a reduction of at least one percentage point, before any run.
Exercises
- EX097 — Design. Design a four-condition experiment for DocSort defences. Rubric: 3 = conditions, fixed factors, seeds, paired tasks, metrics; 2 = four; 1 = vague; 0 = none.
- EX098 — Pitfalls. Pick three pitfalls and say how Rina's design avoids or risks them. Rubric: 3 = three accurate; 2 = two; 1 = one; 0 = none.
- EX099 — Interpretation. Interpret the interval 0.1–1.5 points. Rubric: 3 = likely real but small benefit; may not justify cost alone; 2 = partial; 1 = vague; 0 = wrong.
Synthesis
Research methods II adds rigour: multi-condition paired designs, bootstrap intervals, multiplicity control and a pitfall checklist.
Continue studying — chapter 33 of 40
← Chapter 32 (draft) · Course chapter index · Chapter 34 (draft) →