◇ OpenCyber Academy
Master's · Chapter 3 · English

Chapter 3 — Study design, ethics and reproducibility

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 3 — Study design, ethics and reproducibility

Lesson map

Lesson 03 published. Map, example, EX015.

flowchart TD
  N0["Learning outcomes"]
  N1["3.1 Choosing a design"]
  N2["3.2 Variables and controls"]
  N3["3.3 Ethics and data protection"]
  N4["3.4 Reproducible by design"]
  N5["3.6 Practice activities"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX015"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["3.1 Choosing a design"]
  N2["3.2 Variables and controls"]
  N3["3.3 Ethics and data protection"]
  N4["3.4 Reproducible by design"]
  N5["3.6 Practice activities"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX015"]

[Learning outcomes]
    |
    v
[3.1 Choosing a design]
    |
    v
[3.2 Variables and controls]
    |
    v
[3.3 Ethics and data protection]
    |
    v
[3.4 Reproducible by design]
    |
    v
[3.6 Practice activities]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX015]

Learning outcomes

You will (1) choose a design (controlled experiment, observational study, case study) that fits the question; (2) define variables, controls, randomisation and sample size; (3) plan data handling and ethics for personal or sensitive data; (4) make the study reproducible from the start; and (5) document the plan so a supervisor can approve it.

3.1 Choosing a design

Most master's projects in AI security are controlled experiments: you change one factor (deduplication on/off), hold others fixed, and measure outcomes. Observational studies analyse existing data without intervening (for example, logs of attempted prompt injections against a deployed assistant) and must worry about confounding. Case studies examine one system in depth and are useful when experiments are impossible, but they support weaker generalisation. Pick the strongest design you can run ethically and within time. Leo's question compares conditions he can control, so a controlled experiment fits.

3.2 Variables and controls

For Leo: the independent variable is training-set treatment (original, exact-deduplicated, near-deduplicated). Dependent variables are membership-inference TPR at 1% FPR, AUC, and classifier macro-F1. Controlled variables include model architecture, epochs, learning rate, random seeds and evaluation split. Randomise which records go into training versus held-out sets so that members and non-members come from the same distribution — otherwise an attack may detect distribution shift rather than membership, a known pitfall. Use several seeds and report variability. Estimate sample size from a pilot (Chapter 5).

3.3 Ethics and data protection

When data relates to people, apply data-protection principles: lawful basis, purpose limitation, minimisation, storage limitation and security. The GDPR applies in the EU; other jurisdictions have similar laws. Prefer synthetic data that preserves the properties you need; if real data is essential, obtain approval from your institution's ethics board and the data controller, pseudonymise, restrict access, and set deletion dates. Privacy attacks on real data create the very harm you study; run them only on copies under agreement, and never publish recovered records. If your project involves people (annotators, survey participants), you need informed consent.

3.4 Reproducible by design

Set up from day one: a version-controlled repository; pinned environment; configuration files rather than hard-coded settings; fixed seeds; a single command to run each experiment; logs written as machine-readable records; and a data card describing provenance and processing (Gebru et al., arXiv:1803.09010). The NeurIPS reproducibility programme (Pineau et al., arXiv:2003.12206) offers a checklist worth adopting. Reproducibility is far cheaper to build in than to retrofit. Leo's design lists the repository, a lock file and a run.py --config interface from the start, which made his later reruns simple.

3.5 Worked example (synthetic Harbourline case)

Leo's design document specifies a controlled experiment with three conditions × five seeds. Data: 20,000 synthetic claim notes generated from templates with controlled duplication (8% exact duplicates, 6% near-duplicates), so the ground truth of duplication is known. Members and non-members are drawn by random split from the same pool. The ethics section records that no real claimant data is used, that Harbourline's data-protection officer reviewed the plan, and that the synthetic generator will be released. The reproducibility section lists the repository, environment lock file and a run.py --config interface.

The stats lab shows why five seeds are planned: per-seed results vary even when nothing changes but the seed.

"""Lab S: reporting attack-success rates honestly: Wilson intervals,
seed variation and a paired bootstrap. Synthetic trial outcomes only."""
import math, random
def wilson(k, n, z=1.96):
    p = k/n; d = 1 + z*z/n
    c = (p + z*z/(2*n))/d; h = z*math.sqrt(p*(1-p)/n + z*z/(4*n*n))/d
    return c - h, c + h
for k, n in ((3, 20), (30, 200), (300, 2000)):
    lo, hi = wilson(k, n); print(f"successes={k}/{n} rate={k/n:.3f} wilson95=[{lo:.3f}, {hi:.3f}]")
# five seeds of a synthetic defence evaluation: per-seed attack success rate
rates = []
for seed in range(5):
    r = random.Random(seed); rates.append(sum(r.random() < 0.12 for _ in range(200))/200)
m = sum(rates)/5; sd = math.sqrt(sum((x-m)**2 for x in rates)/4)
print("per-seed rates:", " ".join(f"{x:.3f}" for x in rates), f"mean={m:.3f} sd={sd:.3f}")
# paired bootstrap: baseline vs defence on the same 300 synthetic prompts
r = random.Random(42)
base = [r.random() < 0.30 for _ in range(300)]
defn = [b and r.random() < 0.6 for b in base]   # defence blocks ~40% of successes
diffs = []
for _ in range(2000):
    idx = [r.randrange(300) for _ in range(300)]
    diffs.append(sum(base[i] for i in idx)/300 - sum(defn[i] for i in idx)/300)
diffs.sort()
print(f"baseline_asr={sum(base)/300:.3f} defended_asr={sum(defn)/300:.3f} "
      f"reduction_ci95=[{diffs[50]:.3f}, {diffs[1949]:.3f}]")

Expected output:

successes=3/20 rate=0.150 wilson95=[0.052, 0.360]
successes=30/200 rate=0.150 wilson95=[0.107, 0.206]
successes=300/2000 rate=0.150 wilson95=[0.135, 0.166]
per-seed rates: 0.110 0.130 0.135 0.105 0.145 mean=0.125 sd=0.017
baseline_asr=0.350 defended_asr=0.190 reduction_ci95=[0.120, 0.203]

3.6 Practice activities

  1. Write your design with variables, controls and randomisation. 2. Draft an ethics and data-protection section. 3. Create the repository skeleton with configuration files and a seed policy. Use Leo's design as a model: name independent, dependent and controlled variables, and say how members and non-members are drawn from the same pool. For the ethics text, say what data is used, who reviewed the plan and what will be released. For the repository, include configuration files, a lock file, a seed policy and a single command per experiment. Ask your supervisor to review all three.

3.7 Errors and limits

Four errors recur, and each undermines the result; check your design for all four:

Exercises

Synthesis

Design determines what you can conclude. Controlled variables, matched member/non-member sampling, multiple seeds, ethical data handling and reproducibility built in from the start are what let a short project produce a credible answer.

Continue studying — chapter 03 of 08

← Chapter 02 (draft) · Course chapter index · Chapter 04 (draft) →