Chapter 4 — Baseline replication and pilot
Lesson map
Lesson 04 published. Map, example, EX021.
flowchart TD
N0["Learning outcomes"]
N1["4.1 Why replicate first"]
N2["4.2 A replication protocol"]
N3["4.3 Pilot studies"]
N4["4.4 Harness engineering"]
N5["4.6 Practice activities"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX021"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["4.1 Why replicate first"]
N2["4.2 A replication protocol"]
N3["4.3 Pilot studies"]
N4["4.4 Harness engineering"]
N5["4.6 Practice activities"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX021"]
[Learning outcomes]
|
v
[4.1 Why replicate first]
|
v
[4.2 A replication protocol]
|
v
[4.3 Pilot studies]
|
v
[4.4 Harness engineering]
|
v
[4.6 Practice activities]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX021]
Learning outcomes
You will (1) reproduce a published AI-security result and classify the outcome (reproduced, partially reproduced, not reproduced); (2) document every deviation; (3) run a pilot that estimates variance and sample size; (4) build an evaluation harness that is versioned and deterministic where possible; and (5) decide, from pilot evidence, whether to proceed, revise or stop.
4.1 Why replicate first
Your contribution will be compared with prior work, so you must know what prior work actually does in your environment. Replication catches three common problems: published numbers that depend on undocumented settings, baselines that are weaker than reported, and evaluation code that measures something other than the paper's text implies. Pineau et al. (arXiv:2003.12206) reported on the NeurIPS 2019 reproducibility programme and its checklist; Henderson et al. (arXiv:1709.06560) showed in reinforcement learning that random seeds and implementation details alone can change conclusions. Use precise terms: repeatability (same team, same setup), reproducibility (different team, same artefacts), replicability (different team, new artefacts or data). Say which you attempted.
4.2 A replication protocol
Hosted model APIs complicate replication because the model behind an endpoint can change. Prefer local open-weights checkpoints identified by hash for the main study, and treat API results as time-stamped observations. The five steps below apply to every baseline in the thesis. Steps 1 and 2 make the run identifiable, step 3 gives the comparison point, step 4 explains any difference, and step 5 states the outcome in the terms defined in 4.1. Ana's replication in 4.5 follows them, including the one-factor-at-a-time diagnosis that isolated temperature.
- Freeze the target. Record the paper version, code commit, model checkpoint hashes, dataset version and licence.
- Re-create the environment. Pin library versions; record hardware and numeric settings; note non-determinism sources (GPU kernels, sampling temperature, API model updates).
- Run as published. No changes first. Record the result with intervals across at least three seeds.
- Diagnose gaps. If numbers differ, change one factor at a time and log each run.
- Classify and report. A result is reproduced if your interval overlaps the published value and the qualitative conclusion holds; partially if the direction holds but the size differs materially; not reproduced otherwise. Contact the authors politely before publishing a failure.
4.3 Pilot studies
A pilot is not a small main study; it estimates what you need to design the main study. Measure variance of the primary metric across seeds and tasks, estimate the effect size, check that the harness logs everything required, and time each run. Use the pilot to compute sample size. For a rate near 10% and a target 95% interval half-width of 2 percentage points, the normal approximation gives n ≈ 1.96² × 0.1 × 0.9 / 0.02² ≈ 864 trials per condition. Pilots also reveal metric pathologies: for example, an LLM judge that labels refusals as "successful defence" even when the tool call already happened. Fix the measurement, not the result.
4.4 Harness engineering
An evaluation harness should be deterministic given a seed, write one JSON record per trial (task id, condition, seed, model hash, prompt template version, full transcript reference, tool calls, labels), and keep labels separate from outcomes so graders cannot see condition names. Put the sealed test set behind a separate access step so it cannot be loaded during development. Add canaries: payloads that should always succeed against the undefended baseline; if they fail, the harness or environment is broken. Ana's grader fix is an example.
4.5 Worked example (synthetic MAAL case)
Ana replicated a published structural defence. As published, she measured 4.1% ASR against the paper's payload list (paper: 1.0%); with three seeds the interval was 3.2–5.2%, so not reproduced in size. Diagnosis showed the paper used temperature 0 while the released config used 0.7; at temperature 0 she obtained 1.4% (0.9–2.1%), reproduced under the protocol in 4.2, with the setting change logged as a deviation. Against her adaptive attacks the defence reached 19% ASR. She emailed the authors, who confirmed the temperature setting and thanked her. The pilot (90 tasks per condition) gave a per-condition SD of about 1.7 points across seeds and a separation effect of roughly 12 points, so she fixed the main study at 900 tasks per condition and five seeds, enlarging the sealed evaluation set from the proposal's 600 tasks and logging the change. The pilot also exposed that her grader counted a tool call as blocked if the final answer contained a refusal; she rewrote the grader to read tool-call logs directly.
The poisoning lab illustrates why pilots should measure more than one metric: as backdoor points increase, clean accuracy barely moves while attack success on triggered inputs climbs from 7.5% to 98%. A pilot that tracked only clean accuracy would miss the attack entirely.
"""Lab P: label-flip poisoning and a backdoor trigger on a toy nearest-centroid
classifier (synthetic data, stdlib only). Teaching model only."""
import random
rng = random.Random(11)
def point(label, trigger=False):
cx = 1.0 if label else -1.0
x = [rng.gauss(cx, 0.8), rng.gauss(cx, 0.8), 0.0]
if trigger: x[2] = 5.0 # third feature is the 'trigger'
return x, label
clean = [point(i % 2) for i in range(400)]
test = [point(i % 2) for i in range(400)]
def fit(data):
c = {}
for lab in (0, 1):
pts = [x for x, y in data if y == lab]
c[lab] = [sum(p[i] for p in pts)/len(pts) for i in range(3)]
return c
def pred(c, x):
d = {k: sum((x[i]-v[i])**2 for i in range(3)) for k, v in c.items()}
return min(d, key=d.get)
def accuracy(c, data): return sum(pred(c, x) == y for x, y in data)/len(data)
print("label-flip poisoning (flip a fraction of class-1 labels to 0):")
for frac in (0.0, 0.1, 0.2, 0.3, 0.4):
data = [(x, y) for x, y in clean]; ones = [i for i, (x, y) in enumerate(data) if y == 1]
for i in ones[:int(frac*len(ones))]: data[i] = (data[i][0], 0)
print(f" flipped={frac:.1f} clean_test_accuracy={accuracy(fit(data), test):.3f}")
print("backdoor: add n triggered class-1 points labelled 0")
trig_test = [point(1, trigger=True) for _ in range(200)]
for n in (0, 10, 40, 80):
data = clean + [(point(1, trigger=True)[0], 0) for _ in range(n)]
c = fit(data)
asr = sum(pred(c, x) == 0 for x, _ in trig_test)/len(trig_test)
print(f" poison_points={n:3d} clean_accuracy={accuracy(c, test):.3f} attack_success={asr:.3f}")
Expected output:
label-flip poisoning (flip a fraction of class-1 labels to 0):
flipped=0.0 clean_test_accuracy=0.965
flipped=0.1 clean_test_accuracy=0.965
flipped=0.2 clean_test_accuracy=0.953
flipped=0.3 clean_test_accuracy=0.950
flipped=0.4 clean_test_accuracy=0.945
backdoor: add n triggered class-1 points labelled 0
poison_points= 0 clean_accuracy=0.965 attack_success=0.075
poison_points= 10 clean_accuracy=0.963 attack_success=0.145
poison_points= 40 clean_accuracy=0.963 attack_success=0.685
poison_points= 80 clean_accuracy=0.968 attack_success=0.980
4.6 Practice activities
- Pick one baseline for your thesis and execute steps 1–3 of the protocol. 2. Compute a sample size from your pilot variance. 3. Add canaries and blinded labels to your harness. For the replication, record the paper version, commit, checkpoint hashes and dataset before running anything, then run as published with at least three seeds. Classify the outcome with the protocol's three categories and write down every deviation. For the sample size, use the formula from the pilot discussion with your own variance and target interval. For the harness, confirm that canaries succeed against the undefended baseline and that graders cannot see condition names.
4.7 Errors and limits
Each of these errors corrupts evidence:
- Changing several settings at once while diagnosing a gap; that would have hidden that temperature alone explained Ana's gap.
- Treating a failed replication as proof the original was wrong without contacting the authors; the failure may come from your environment, so contact them first.
- Using a hosted API endpoint as a fixed artefact; it can change model without notice, so its results are time-stamped observations.
- Letting graders see condition names, so that they may favour the defence.
- Pilot data leaking into the sealed test set, which makes the main study partly tuned on its own test.
Exercises
- EX021 — Terms. Define repeatability, reproducibility and replicability in one line each. Rubric: 3 = all three correct; 2 = two; 1 = one; 0 = none.
- EX022 — Classify. Published ASR 1.0%; your three-seed interval 0.9–2.1% with the same conclusion. Classify and justify. Rubric: 3 = reproduced or partially reproduced with justified reasoning about overlap and size; 2 = label only; 1 = wrong but reasoned; 0 = none.
- EX023 — Sample size. Compute n per condition for a rate near 20% and half-width 3 points (95%). Rubric: 3 = ≈683 with formula; 2 = arithmetic slip; 1 = formula only; 0 = none.
- EX024 — Lab reading. In the poisoning lab, what is clean accuracy and attack success with 40 poison points, and what does it teach about monitoring? Rubric: 3 = 0.963 and 0.685; clean metrics hide backdoors; 2 = numbers only; 1 = vague; 0 = wrong.
- EX025 — Canaries. Explain what an evaluation canary is and what it means if one fails. Rubric: 3 = known-outcome trial; failure indicates harness/environment fault, halt; 2 = definition only; 1 = vague; 0 = wrong.
- EX026 — Grader bug. Why is counting refusals in the final answer an invalid measure of tool-call blocking? Rubric: 3 = the harmful side effect may already have occurred; measure actions from logs; 2 = partial; 1 = vague; 0 = wrong.
Synthesis
Replication tells you what prior work really does in your hands; the pilot tells you how big the main study must be and whether your measurements are sound. Both are evidence in their own right and belong in the thesis.
Continue studying — chapter 04 of 08
← Chapter 03 (draft) · Course chapter index · Chapter 05 (draft) →