◇ OpenCyber Academy
Doctoral research · Chapter 5 · English

Chapter 5 — Main study and independent validation

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 5 — Main study and independent validation

Lesson map

Lesson 05 published. Map, example, EX027.

flowchart TD
  N0["Learning outcomes"]
  N1["5.1 Executing a pre-registered study"]
  N2["5.2 Adaptive attack under a fair budget"]
  N3["5.3 Analysis"]
  N4["5.4 Independent validation"]
  N5["5.6 Practice activities"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX027"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["5.1 Executing a pre-registered study"]
  N2["5.2 Adaptive attack under a fair budget"]
  N3["5.3 Analysis"]
  N4["5.4 Independent validation"]
  N5["5.6 Practice activities"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX027"]

[Learning outcomes]
    |
    v
[5.1 Executing a pre-registered study]
    |
    v
[5.2 Adaptive attack under a fair budget]
    |
    v
[5.3 Analysis]
    |
    v
[5.4 Independent validation]
    |
    v
[5.6 Practice activities]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX027]

Learning outcomes

You will (1) run a pre-registered main study and keep a deviation log; (2) evaluate a defence under adaptive attack with a fair budget; (3) analyse results with paired comparisons, intervals and correction for multiple tests; (4) commission an independent validation (red team, second implementation or external replication); and (5) report what the evidence supports and what it does not.

5.1 Executing a pre-registered study

Freeze the protocol: threat model version, hypotheses, metrics, sample sizes, seeds, model hashes, prompt templates, attack budget and the analysis script. Store this as a timestamped document — a dated commit in a repository or a registration with your institution or a public registry is enough for most venues. Run the study exactly as frozen. Every change (a crashed node, a model checkpoint that would not load, a task template bug) goes into a deviation log with date, reason and expected effect. Deviations are normal; hiding them is not.

5.2 Adaptive attack under a fair budget

The main threat to validity in defence papers is a weak adversary. For each defence, run (a) the static attacks used for baselines; (b) attacks that target the defence mechanism explicitly — for a separation defence, payloads that imitate the reserved instruction channel's delimiters, payloads that ask the model to repeat content into a privileged position, multi-step payloads that use an allowed tool to stage a disallowed one; and (c) an optimisation-based or search-based attack where the threat model permits it. Give every defence the same budget (queries, payload variants, compute) and report attack success as a function of budget, not only at one point: a defence that holds at 20 attempts but fails at 200 has a different security meaning from one that holds at both. The guidance in arXiv:1902.06705 — check that more attack effort never lowers success, and that unbounded attacks reach 100% where they should — applies directly.

5.3 Analysis

Use paired comparisons when conditions share tasks: the same 900 tasks under baseline and defence. The paired bootstrap from Chapter 1 estimates the reduction in ASR with an interval. When you test several hypotheses or several attack families, control false discoveries — Holm–Bonferroni is simple and conservative. Report effect sizes with intervals, utility costs on the same tasks, and per-family results so that an average does not hide a family where the defence fails. Stratify by model: a defence that helps one model and harms another needs both numbers in the abstract, not one pooled number.

5.4 Independent validation

Doctoral evidence is stronger when someone other than the author tries to break or reproduce it. Three forms are practical. A red-team validation: give a separate team the defence, documentation and budget, but not the sealed test tasks, and ask for maximum ASR. A second implementation: another person implements the method from the paper text alone; differences reveal undocumented choices. An external replication: a collaborator in another lab runs your released artefacts. Agree in advance how results will be reported, including if the red team wins. The MITRE ATLAS matrix is a useful checklist for red-team coverage, and NIST AI 600-1 lists red-teaming among suggested actions for generative AI risk management.

5.5 Worked example (synthetic MAAL case)

Ana's frozen protocol ran 900 tasks × 4 conditions × 5 seeds on two local models. Deviation log: one checkpoint failed to load on seed 3 and was re-run after a driver fix (no expected effect). Results (synthetic): under static payloads, ASR was 31% (no defence), 18% (filter), 6% (allowlist), 2.1% (separation + allowlist). Under adaptive attack at 200 variants per task family, ASR became 44%, 37%, 11% and 5.8%. The paired reduction of separation + allowlist versus allowlist alone was 5.2 points (95% CI 3.9–6.6) and survived Holm correction. Task success dropped by 2.4 points (H2 met). The independent red team, with the same budget, reached 8.9% on one model by chaining an allowed search tool into a summary that echoed instructions — a family Ana had not anticipated. She did not discard it: it became a named limitation and a new attack family in the paper.

The membership-inference lab is a reminder that independent validation also applies to measurement: the "attack" signal (AUC 0.60 at k = 1) nearly disappears at k = 25. Whether your audit finds leakage depends as much on the model configuration as on the attack.

"""Lab M: membership inference against an overfit 1-nearest-neighbour 'model'
that returns a confidence score. Synthetic records only; teaching model."""
import random, math
rng = random.Random(5)
def rec(): 
    y = rng.random() < 0.5
    return [rng.gauss(0.6 if y else -0.6, 1.0) for _ in range(4)], int(y)
members = [rec() for _ in range(150)]; nonmembers = [rec() for _ in range(150)]
def confidence(x, y, k):
    # k-NN vote share for the true label, computed over the training set
    d = sorted((sum((a-b)**2 for a, b in zip(x, m)), my) for m, my in members)
    return sum(1 for _, my in d[:k] if my == y)/k
def auc(pos, neg):
    s = sum((p > n) + 0.5*(p == n) for p in pos for n in neg)
    return s/(len(pos)*len(neg))
for k in (1, 5, 25):
    pos = [confidence(x, y, k) for x, y in members]
    neg = [confidence(x, y, k) for x, y in nonmembers]
    best = max(sum(p >= t for p in pos)/len(pos) - sum(n >= t for n in neg)/len(neg)
               for t in sorted(set(pos + neg)))
    print(f"k={k:2d} attack_auc={auc(pos, neg):.3f} best_advantage={best:.3f}")

Expected output:

k= 1 attack_auc=0.600 best_advantage=0.200
k= 5 attack_auc=0.521 best_advantage=0.073
k=25 attack_auc=0.520 best_advantage=0.060

5.6 Practice activities

  1. Freeze your protocol and open a deviation log. 2. Write three adaptive attacks that target your defence's mechanism. 3. Draft a red-team brief with budget, scope, rules of engagement and reporting terms. Your protocol should list threat model version, hypotheses, metrics, sample sizes, seeds, model hashes, templates, attack budget and the analysis script, with a date. The adaptive attacks should target how your defence works, as Ana's delimiter-imitating and repetition payloads do. The red-team brief should give the defence and budget but not the sealed tasks, and say in advance how results will be reported if the red team wins.

5.7 Errors and limits

Each of these errors overstates the defence:

Exercises

Synthesis

A main study earns trust through a frozen protocol, a fair and adaptive adversary, paired and corrected analysis, and someone else's attempt to break it. What the red team finds is part of the contribution.

Continue studying — chapter 05 of 08

← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →