Chapter 1 — Advanced research integrity and methods
Lesson map
Lesson 01 published. Map, example, EX001.
flowchart TD
N0["Learning outcomes"]
N1["1.1 Why integrity is a method, not a courtesy"]
N2["1.2 Security claims and capability claims"]
N3["1.3 Methods that survive review"]
N4["1.4 Authorization, data and disclosure"]
N5["1.6 Practice activities"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX001"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["1.1 Why integrity is a method, not a courtesy"]
N2["1.2 Security claims and capability claims"]
N3["1.3 Methods that survive review"]
N4["1.4 Authorization, data and disclosure"]
N5["1.6 Practice activities"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX001"]
[Learning outcomes]
|
v
[1.1 Why integrity is a method, not a courtesy]
|
v
[1.2 Security claims and capability claims]
|
v
[1.3 Methods that survive review]
|
v
[1.4 Authorization, data and disclosure]
|
v
[1.6 Practice activities]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX001]
Learning outcomes
You will (1) explain why AI-security research is unusually exposed to measurement error, leakage and dual-use harm; (2) separate security claims from capability claims and state each so it can be falsified; (3) design an evaluation with a declared threat model, pre-specified metrics and an adaptive adversary; (4) apply the rules on authorization, data protection and responsible disclosure that bind security research on models and AI services; and (5) report uncertainty with intervals rather than single numbers.
1.1 Why integrity is a method, not a courtesy
Research integrity in AI security is not a list of polite habits; it is the set of procedures that make a result true outside the laboratory where it was produced. The field has a documented record of results that did not survive contact with a stronger adversary. Athalye, Carlini and Wagner (arXiv:1802.00420) showed that seven of nine defences accepted at a leading conference relied on obfuscated gradients: the defence did not make the model robust, it only broke the attacker's optimisation. Once the attack was adapted, most defences fell. Carlini and colleagues then wrote On Evaluating Adversarial Robustness (arXiv:1902.06705), a checklist that turns that lesson into method: state the threat model, use adaptive attacks, report the attack budget, verify that stronger attacks never do worse than weaker ones and release code. Arp et al. (arXiv:2010.09470) catalogued ten pitfalls of machine learning in computer security — sampling bias, label inaccuracy, data snooping, spurious correlations, biased parameter selection, inappropriate baselines, base-rate fallacy, lab-only evaluation, inappropriate threat model and inappropriate performance measures — and found them widespread in top-venue papers. Kapoor and Narayanan (arXiv:2207.07048) showed that leakage between training and evaluation data inflated results across many scientific fields that used machine learning. Lipton and Steinhardt (arXiv:1807.03341) described broader "troubling trends": failure to distinguish explanation from speculation, unclear sources of gains and misuse of mathematics.
A doctoral researcher should treat these papers as a description of the default failure mode. If you do nothing special, your result will probably be optimistic.
1.2 Security claims and capability claims
A capability claim says what a system can do ("the detector flags 94% of injected documents"). A security claim says what an adversary cannot do under stated conditions ("no adaptive attacker with black-box access and 1,000 queries achieves more than 5% tool hijacking"). Security claims are universally quantified over adversaries, which is why they are fragile: one counter-example refutes them. Write every claim in three parts: the adversary (goals, knowledge, capabilities, budget), the system (model, version, configuration, tools, data) and the property with its metric and threshold. NIST AI 100-2 E2025 organises attacks on predictive and generative AI by attacker goal (availability, integrity, privacy, and for generative systems misuse enablement), knowledge (white-, grey-, black-box), capability (training-data control, model control, query access, resource control) and lifecycle stage; use its vocabulary so reviewers can place your claim.
1.3 Methods that survive review
Five habits carry most of the weight. Pre-specification: write hypotheses, metrics, datasets and stopping rules before you look at test results, and log every change after. Separation: keep a sealed test set that no tuning ever touches; deduplicate near-identical items across splits, which matters greatly for language data scraped from the web. Adaptive adversaries: for every defence, ask how an attacker who knows the defence would respond, and build that attack. Strong baselines: compare against the best simple method, not a straw man. Uncertainty: report confidence intervals, number of seeds and variance. A 3-out-of-20 attack success rate (15%) has a 95% Wilson interval of roughly 5% to 36% — far too wide to rank two defences.
"""Lab S: reporting attack-success rates honestly: Wilson intervals,
seed variation and a paired bootstrap. Synthetic trial outcomes only."""
import math, random
def wilson(k, n, z=1.96):
p = k/n; d = 1 + z*z/n
c = (p + z*z/(2*n))/d; h = z*math.sqrt(p*(1-p)/n + z*z/(4*n*n))/d
return c - h, c + h
for k, n in ((3, 20), (30, 200), (300, 2000)):
lo, hi = wilson(k, n); print(f"successes={k}/{n} rate={k/n:.3f} wilson95=[{lo:.3f}, {hi:.3f}]")
# five seeds of a synthetic defence evaluation: per-seed attack success rate
rates = []
for seed in range(5):
r = random.Random(seed); rates.append(sum(r.random() < 0.12 for _ in range(200))/200)
m = sum(rates)/5; sd = math.sqrt(sum((x-m)**2 for x in rates)/4)
print("per-seed rates:", " ".join(f"{x:.3f}" for x in rates), f"mean={m:.3f} sd={sd:.3f}")
# paired bootstrap: baseline vs defence on the same 300 synthetic prompts
r = random.Random(42)
base = [r.random() < 0.30 for _ in range(300)]
defn = [b and r.random() < 0.6 for b in base] # defence blocks ~40% of successes
diffs = []
for _ in range(2000):
idx = [r.randrange(300) for _ in range(300)]
diffs.append(sum(base[i] for i in idx)/300 - sum(defn[i] for i in idx)/300)
diffs.sort()
print(f"baseline_asr={sum(base)/300:.3f} defended_asr={sum(defn)/300:.3f} "
f"reduction_ci95=[{diffs[50]:.3f}, {diffs[1949]:.3f}]")
Expected output:
successes=3/20 rate=0.150 wilson95=[0.052, 0.360]
successes=30/200 rate=0.150 wilson95=[0.107, 0.206]
successes=300/2000 rate=0.150 wilson95=[0.135, 0.166]
per-seed rates: 0.110 0.130 0.135 0.105 0.145 mean=0.125 sd=0.017
baseline_asr=0.350 defended_asr=0.190 reduction_ci95=[0.120, 0.203]
The lab shows the same 15% rate at three sample sizes. Only the largest sample supports a two-decimal claim. Across five seeds the rate moves between 10.5% and 14.5%, so a single-seed result could mislead. The paired bootstrap estimates the reduction a defence achieves on the same prompts, which is the comparison you actually care about, and its interval excludes zero.
1.4 Authorization, data and disclosure
Security research on AI touches other people's systems, data and safety. Test only systems you own or are authorised in writing to test, with a scope that names models, endpoints, accounts, rate limits and dates. Terms of service of hosted model APIs typically restrict some forms of testing; read them and ask the provider when in doubt. Personal data in prompts, training sets or logs falls under data-protection law (for example the GDPR in the EU); minimise, pseudonymise and get ethics-board approval when human participants or their data are involved. When you find a weakness in a deployed product, use coordinated disclosure: notify the vendor privately, agree a timeline, and publish without operational detail that would mainly help attackers. Dual-use review should be explicit: for each artefact (code, prompts, datasets, model weights) ask who benefits more from release, defenders or attackers, and whether a staged or gated release changes the answer.
1.5 Worked example (synthetic MAAL case)
Ana's first draft claim was "our separation defence stops prompt injection". Her supervisor rewrote it: Adversary: a third party who controls the text of up to one retrieved document per task, knows the defence design (Kerckhoffs-style), has no model weights, and may try 200 payload variants per task family. System: a local open-weights model in a sandbox with three tools (search, summarise, email-to-allowlisted-addresses), fixed prompt template v1.3. Property: rate of unauthorised tool calls (any call not requested by the user) over 600 held-out tasks, with a 95% interval upper bound below 2%. The rewrite is falsifiable, names a budget and binds the claim to a version. It also exposes the hard part: "knows the defence design" means Ana must build adaptive attacks against her own method.
1.6 Practice activities
- Take a recent AI-security paper you admire and rewrite its main claim in adversary–system–property form. 2. Run the Arp et al. pitfall list against your own last experiment. 3. Draft a one-page authorization and data plan for your doctoral testbed. For the first activity, use the MAAL rewrite as a model: name the adversary's knowledge and budget, bind the system to a version, and give the property a metric and threshold. If the paper does not say enough to fill a part, record that as a gap in the paper. For the second, mark each of the ten pitfalls as present, absent or unknown, with a sentence of evidence. For the third, name models, endpoints, data and dates, and say who approves.
1.7 Errors and limits
Each of these errors makes a result look stronger than it is:
- Reporting the best of many runs as "the" result, which hides the spread that Lab S shows across seeds.
- Tuning prompts or thresholds on the test set, which turns it into training data.
- Evaluating a defence only against attacks designed before it existed; such a defence has not met the adaptive adversary in Ana's threat model.
- Treating "no attack found" as proof of security, when it may only mean the search was weak.
- Publishing working exploit strings against a live product before the vendor can respond, which harms users before a fix exists.
Exercises
- EX001 — Claim rewrite. Rewrite "our filter makes chatbots safe from jailbreaks" as a falsifiable security claim. Rubric: 3 = adversary with knowledge and budget, system with version, property with metric and threshold; 2 = one part vague; 1 = two parts missing; 0 = unchanged or unsafe.
- EX002 — Obfuscated gradients. In two sentences, explain why a defence that breaks the attacker's gradient computation is not evidence of robustness. Rubric: 3 = names attack failure versus model robustness and the remedy (adaptive/BPDA-style or gradient-free attack); 2 = idea without remedy; 1 = vague; 0 = wrong.
- EX003 — Interval. Using the lab, state what you can and cannot claim from 3/20 successes. Rubric: 3 = interval ≈0.05–0.36, cannot rank defences with a few points' difference, need larger n; 2 = interval only; 1 = point estimate only; 0 = wrong.
- EX004 — Pitfalls. Name three of the ten pitfalls from Arp et al. and give a guard for each. Rubric: 3 = three correct with guards; 2 = three without guards; 1 = one or two; 0 = none.
- EX005 — Leakage. Describe how near-duplicate prompts across train and test splits inflate a jailbreak detector's score, and one test for it. Rubric: 3 = mechanism plus dedup/hash or n-gram overlap test; 2 = mechanism only; 1 = vague; 0 = wrong.
- EX006 — Scope. List five fields an authorization for testing a hosted model must contain. Rubric: 3 = systems/endpoints, methods, dates, rate/data limits, contacts and stop conditions; 2 = four; 1 = two or three; 0 = none or unsafe.
- EX007 — Dual use. For a dataset of 5,000 successful injection payloads, propose a release decision and justify it. Rubric: 3 = weighs defender and attacker benefit, proposes gated/staged release or redaction, notes vendor disclosure; 2 = decision with partial reasons; 1 = decision only; 0 = unconditional public release of live exploits with no reasoning.
Synthesis
Integrity in AI security is operational: falsifiable claims, sealed test data, adaptive adversaries, honest uncertainty and authorised, disclosed work. These are the tools that make the rest of the doctorate trustworthy.