## Chapter 1 — Advanced research integrity and methods ### Learning outcomes You will (1) explain why AI-security research is unusually exposed to measurement error, leakage and dual-use harm; (2) separate *security claims* from *capability claims* and state each so it can be falsified; (3) design an evaluation with a declared threat model, pre-specified metrics and an adaptive adversary; (4) apply the rules on authorization, data protection and responsible disclosure that bind security research on models and AI services; and (5) report uncertainty with intervals rather than single numbers. ### 1.1 Why integrity is a method, not a courtesy Research integrity in AI security is not a list of polite habits; it is the set of procedures that make a result true outside the laboratory where it was produced. The field has a documented record of results that did not survive contact with a stronger adversary. Athalye, Carlini and Wagner (arXiv:1802.00420) showed that seven of nine defences accepted at a leading conference relied on *obfuscated gradients*: the defence did not make the model robust, it only broke the attacker's optimisation. Once the attack was adapted, most defences fell. Carlini and colleagues then wrote *On Evaluating Adversarial Robustness* (arXiv:1902.06705), a checklist that turns that lesson into method: state the threat model, use adaptive attacks, report the attack budget, verify that stronger attacks never do worse than weaker ones and release code. Arp et al. (arXiv:2010.09470) catalogued ten pitfalls of machine learning in computer security — sampling bias, label inaccuracy, data snooping, spurious correlations, biased parameter selection, inappropriate baselines, base-rate fallacy, lab-only evaluation, inappropriate threat model and inappropriate performance measures — and found them widespread in top-venue papers. Kapoor and Narayanan (arXiv:2207.07048) showed that *leakage* between training and evaluation data inflated results across many scientific fields that used machine learning. Lipton and Steinhardt (arXiv:1807.03341) described broader "troubling trends": failure to distinguish explanation from speculation, unclear sources of gains and misuse of mathematics. A doctoral researcher should treat these papers as a description of the default failure mode. If you do nothing special, your result will probably be optimistic. ### 1.2 Security claims and capability claims A *capability claim* says what a system can do ("the detector flags 94% of injected documents"). A *security claim* says what an adversary cannot do under stated conditions ("no adaptive attacker with black-box access and 1,000 queries achieves more than 5% tool hijacking"). Security claims are universally quantified over adversaries, which is why they are fragile: one counter-example refutes them. Write every claim in three parts: the **adversary** (goals, knowledge, capabilities, budget), the **system** (model, version, configuration, tools, data) and the **property** with its metric and threshold. NIST AI 100-2 E2025 organises attacks on predictive and generative AI by attacker goal (availability, integrity, privacy, and for generative systems misuse enablement), knowledge (white-, grey-, black-box), capability (training-data control, model control, query access, resource control) and lifecycle stage; use its vocabulary so reviewers can place your claim. ### 1.3 Methods that survive review Five habits carry most of the weight. **Pre-specification**: write hypotheses, metrics, datasets and stopping rules before you look at test results, and log every change after. **Separation**: keep a sealed test set that no tuning ever touches; deduplicate near-identical items across splits, which matters greatly for language data scraped from the web. **Adaptive adversaries**: for every defence, ask how an attacker who knows the defence would respond, and build that attack. **Strong baselines**: compare against the best simple method, not a straw man. **Uncertainty**: report confidence intervals, number of seeds and variance. A 3-out-of-20 attack success rate (15%) has a 95% Wilson interval of roughly 5% to 36% — far too wide to rank two defences. ```python """Lab S: reporting attack-success rates honestly: Wilson intervals, seed variation and a paired bootstrap. Synthetic trial outcomes only.""" import math, random def wilson(k, n, z=1.96): p = k/n; d = 1 + z*z/n c = (p + z*z/(2*n))/d; h = z*math.sqrt(p*(1-p)/n + z*z/(4*n*n))/d return c - h, c + h for k, n in ((3, 20), (30, 200), (300, 2000)): lo, hi = wilson(k, n); print(f"successes={k}/{n} rate={k/n:.3f} wilson95=[{lo:.3f}, {hi:.3f}]") # five seeds of a synthetic defence evaluation: per-seed attack success rate rates = [] for seed in range(5): r = random.Random(seed); rates.append(sum(r.random() < 0.12 for _ in range(200))/200) m = sum(rates)/5; sd = math.sqrt(sum((x-m)**2 for x in rates)/4) print("per-seed rates:", " ".join(f"{x:.3f}" for x in rates), f"mean={m:.3f} sd={sd:.3f}") # paired bootstrap: baseline vs defence on the same 300 synthetic prompts r = random.Random(42) base = [r.random() < 0.30 for _ in range(300)] defn = [b and r.random() < 0.6 for b in base] # defence blocks ~40% of successes diffs = [] for _ in range(2000): idx = [r.randrange(300) for _ in range(300)] diffs.append(sum(base[i] for i in idx)/300 - sum(defn[i] for i in idx)/300) diffs.sort() print(f"baseline_asr={sum(base)/300:.3f} defended_asr={sum(defn)/300:.3f} " f"reduction_ci95=[{diffs[50]:.3f}, {diffs[1949]:.3f}]") ``` Expected output: ```text successes=3/20 rate=0.150 wilson95=[0.052, 0.360] successes=30/200 rate=0.150 wilson95=[0.107, 0.206] successes=300/2000 rate=0.150 wilson95=[0.135, 0.166] per-seed rates: 0.110 0.130 0.135 0.105 0.145 mean=0.125 sd=0.017 baseline_asr=0.350 defended_asr=0.190 reduction_ci95=[0.120, 0.203] ``` The lab shows the same 15% rate at three sample sizes. Only the largest sample supports a two-decimal claim. Across five seeds the rate moves between 10.5% and 14.5%, so a single-seed result could mislead. The paired bootstrap estimates the reduction a defence achieves on the *same* prompts, which is the comparison you actually care about, and its interval excludes zero. ### 1.4 Authorization, data and disclosure Security research on AI touches other people's systems, data and safety. Test only systems you own or are authorised in writing to test, with a scope that names models, endpoints, accounts, rate limits and dates. Terms of service of hosted model APIs typically restrict some forms of testing; read them and ask the provider when in doubt. Personal data in prompts, training sets or logs falls under data-protection law (for example the GDPR in the EU); minimise, pseudonymise and get ethics-board approval when human participants or their data are involved. When you find a weakness in a deployed product, use coordinated disclosure: notify the vendor privately, agree a timeline, and publish without operational detail that would mainly help attackers. Dual-use review should be explicit: for each artefact (code, prompts, datasets, model weights) ask who benefits more from release, defenders or attackers, and whether a staged or gated release changes the answer. ### 1.5 Worked example (synthetic MAAL case) Ana's first draft claim was "our separation defence stops prompt injection". Her supervisor rewrote it: *Adversary*: a third party who controls the text of up to one retrieved document per task, knows the defence design (Kerckhoffs-style), has no model weights, and may try 200 payload variants per task family. *System*: a local open-weights model in a sandbox with three tools (search, summarise, email-to-allowlisted-addresses), fixed prompt template v1.3. *Property*: rate of unauthorised tool calls (any call not requested by the user) over 600 held-out tasks, with a 95% interval upper bound below 2%. The rewrite is falsifiable, names a budget and binds the claim to a version. It also exposes the hard part: "knows the defence design" means Ana must build adaptive attacks against her own method. ### 1.6 Practice activities 1. Take a recent AI-security paper you admire and rewrite its main claim in adversary–system–property form. 2. Run the Arp et al. pitfall list against your own last experiment. 3. Draft a one-page authorization and data plan for your doctoral testbed. For the first activity, use the MAAL rewrite as a model: name the adversary's knowledge and budget, bind the system to a version, and give the property a metric and threshold. If the paper does not say enough to fill a part, record that as a gap in the paper. For the second, mark each of the ten pitfalls as present, absent or unknown, with a sentence of evidence. For the third, name models, endpoints, data and dates, and say who approves. ### 1.7 Errors and limits Each of these errors makes a result look stronger than it is: - Reporting the best of many runs as "the" result, which hides the spread that Lab S shows across seeds. - Tuning prompts or thresholds on the test set, which turns it into training data. - Evaluating a defence only against attacks designed before it existed; such a defence has not met the adaptive adversary in Ana's threat model. - Treating "no attack found" as proof of security, when it may only mean the search was weak. - Publishing working exploit strings against a live product before the vendor can respond, which harms users before a fix exists. ### Exercises - **EX001 — Claim rewrite.** Rewrite "our filter makes chatbots safe from jailbreaks" as a falsifiable security claim. Rubric: 3 = adversary with knowledge and budget, system with version, property with metric and threshold; 2 = one part vague; 1 = two parts missing; 0 = unchanged or unsafe. - **EX002 — Obfuscated gradients.** In two sentences, explain why a defence that breaks the attacker's gradient computation is not evidence of robustness. Rubric: 3 = names attack failure versus model robustness and the remedy (adaptive/BPDA-style or gradient-free attack); 2 = idea without remedy; 1 = vague; 0 = wrong. - **EX003 — Interval.** Using the lab, state what you can and cannot claim from 3/20 successes. Rubric: 3 = interval ≈0.05–0.36, cannot rank defences with a few points' difference, need larger n; 2 = interval only; 1 = point estimate only; 0 = wrong. - **EX004 — Pitfalls.** Name three of the ten pitfalls from Arp et al. and give a guard for each. Rubric: 3 = three correct with guards; 2 = three without guards; 1 = one or two; 0 = none. - **EX005 — Leakage.** Describe how near-duplicate prompts across train and test splits inflate a jailbreak detector's score, and one test for it. Rubric: 3 = mechanism plus dedup/hash or n-gram overlap test; 2 = mechanism only; 1 = vague; 0 = wrong. - **EX006 — Scope.** List five fields an authorization for testing a hosted model must contain. Rubric: 3 = systems/endpoints, methods, dates, rate/data limits, contacts and stop conditions; 2 = four; 1 = two or three; 0 = none or unsafe. - **EX007 — Dual use.** For a dataset of 5,000 successful injection payloads, propose a release decision and justify it. Rubric: 3 = weighs defender and attacker benefit, proposes gated/staged release or redaction, notes vendor disclosure; 2 = decision with partial reasons; 1 = decision only; 0 = unconditional public release of live exploits with no reasoning. ### Synthesis Integrity in AI security is operational: falsifiable claims, sealed test data, adaptive adversaries, honest uncertainty and authorised, disclosed work. These are the tools that make the rest of the doctorate trustworthy. ### Continue studying — chapter 01 of 08 [← Course home](../../README.md) · [Course chapter index](../SUBJECTS.md) · [Chapter 02 (draft) →](#ch-02) ## Worked answers **EX001.** "Against an adversary who controls user prompts, knows the filter design and may submit 500 attempts per policy category, model *M* v2.1 with filter *F* v1.0 produces policy-violating output on fewer than 3% of 1,000 held-out harmful requests (upper 95% bound), with a false-refusal rate below 5% on 1,000 benign requests." **EX002.** A defence that breaks gradient computation makes the gradient-based attack fail, not the model correct; the same inputs may still be misclassified. The remedy is an adaptive attack — approximate the broken component (BPDA), use expectation over transformations, or switch to gradient-free search. **EX003.** 3/20 = 15% with a 95% Wilson interval of about 5.2% to 36.0%. You can say the rate is probably not zero and probably below 40%; you cannot rank it against another defence a few points away. Increase n (2,000 trials give about 13.5%–16.6%). **EX004.** Any three, e.g. data snooping → seal test data and pre-register; base-rate fallacy → report precision and FPR at realistic base rates; lab-only evaluation → validate in a realistic deployment or state the limit; inappropriate baseline → include strong simple baselines; sampling bias → document and diversify data sources. **EX005.** If paraphrases or near-copies of a test prompt are in training, the detector recognises memorised text, not the attack concept, so scores rise. Test: hash normalised text and compute n-gram (e.g., 8-gram) or embedding similarity across splits; remove or group duplicates before splitting. **EX006.** Systems and endpoints in scope (model versions, accounts); permitted methods; dates and times; rate, data and content limits (no personal data, no live third parties); contacts, stop conditions and reporting route. Signed by someone with authority. **EX007.** Gate it. Defenders gain evaluation data, but working payloads lower the cost of attacks against live systems. Release a redacted or synthetic-canary version publicly, give the full corpus to vetted researchers under terms, and disclose product-specific findings to vendors first.