◇ OpenCyber Academy
Master's · Chapter 6 · English

Chapter 6 — Data collection and analysis

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 6 — Data collection and analysis

Lesson map

Lesson 06 published. Map, example, EX033.

flowchart TD
  N0["Learning outcomes"]
  N1["6.1 Running the main study"]
  N2["6.2 Data-quality checks"]
  N3["6.3 Primary analysis"]
  N4["6.4 Secondary outcomes and costs"]
  N5["6.6 Practice activities"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX033"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["6.1 Running the main study"]
  N2["6.2 Data-quality checks"]
  N3["6.3 Primary analysis"]
  N4["6.4 Secondary outcomes and costs"]
  N5["6.6 Practice activities"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX033"]

[Learning outcomes]
    |
    v
[6.1 Running the main study]
    |
    v
[6.2 Data-quality checks]
    |
    v
[6.3 Primary analysis]
    |
    v
[6.4 Secondary outcomes and costs]
    |
    v
[6.6 Practice activities]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX033]

Learning outcomes

You will (1) run the main experiments exactly as planned and log deviations; (2) check data quality before analysis; (3) compute primary outcomes with uncertainty (Wilson intervals, bootstrap, variation across seeds); (4) analyse paired comparisons and secondary outcomes; and (5) present results in tables and figures that answer the research question.

6.1 Running the main study

Freeze code at a tagged commit, lock the configuration files, and run conditions in an interleaved or randomised order so that time-dependent problems (a node slowing down, a library update) do not align with one condition. Each run writes a record: condition, seed, config hash, start and end time, metrics and paths to raw outputs. Keep a deviation log as in the pilot. Leo's runs were interleaved by condition and seed, and his deviation log recorded the missing seed-4 run and its rerun. Check the log before analysis, not after.

6.2 Data-quality checks

Before computing anything, check: are all expected runs present? Do canary checks pass (the undefended model must be attackable; a model trained on shuffled labels should perform at chance)? Are there duplicates across splits? Are metric values within plausible ranges? These checks catch bugs that would otherwise become "findings". Leo's checks found one missing seed-4 run, which was rerun and logged before analysis. Write the checks as a script so they run the same way each time, and keep their output with the results.

6.3 Primary analysis

Compute the primary outcome per condition with intervals. For rates, use Wilson intervals; for metrics averaged over items, bootstrap over items; report variation across seeds separately. For paired comparisons (same records or prompts across conditions), bootstrap the difference. Follow the analysis plan; label any extra analysis as exploratory. When several comparisons are made, correct for multiplicity or present them as exploratory. The lab below demonstrates the three tools. With 3 of 20 successes, the Wilson interval spans 5% to 36%: such a small sample cannot support a precise claim. The per-seed rates show seed-to-seed spread. The paired bootstrap estimates the reduction of a defence on the same prompts with an interval excluding zero.

"""Lab S: reporting attack-success rates honestly: Wilson intervals,
seed variation and a paired bootstrap. Synthetic trial outcomes only."""
import math, random
def wilson(k, n, z=1.96):
    p = k/n; d = 1 + z*z/n
    c = (p + z*z/(2*n))/d; h = z*math.sqrt(p*(1-p)/n + z*z/(4*n*n))/d
    return c - h, c + h
for k, n in ((3, 20), (30, 200), (300, 2000)):
    lo, hi = wilson(k, n); print(f"successes={k}/{n} rate={k/n:.3f} wilson95=[{lo:.3f}, {hi:.3f}]")
# five seeds of a synthetic defence evaluation: per-seed attack success rate
rates = []
for seed in range(5):
    r = random.Random(seed); rates.append(sum(r.random() < 0.12 for _ in range(200))/200)
m = sum(rates)/5; sd = math.sqrt(sum((x-m)**2 for x in rates)/4)
print("per-seed rates:", " ".join(f"{x:.3f}" for x in rates), f"mean={m:.3f} sd={sd:.3f}")
# paired bootstrap: baseline vs defence on the same 300 synthetic prompts
r = random.Random(42)
base = [r.random() < 0.30 for _ in range(300)]
defn = [b and r.random() < 0.6 for b in base]   # defence blocks ~40% of successes
diffs = []
for _ in range(2000):
    idx = [r.randrange(300) for _ in range(300)]
    diffs.append(sum(base[i] for i in idx)/300 - sum(defn[i] for i in idx)/300)
diffs.sort()
print(f"baseline_asr={sum(base)/300:.3f} defended_asr={sum(defn)/300:.3f} "
      f"reduction_ci95=[{diffs[50]:.3f}, {diffs[1949]:.3f}]")

Expected output:

successes=3/20 rate=0.150 wilson95=[0.052, 0.360]
successes=30/200 rate=0.150 wilson95=[0.107, 0.206]
successes=300/2000 rate=0.150 wilson95=[0.135, 0.166]
per-seed rates: 0.110 0.130 0.135 0.105 0.145 mean=0.125 sd=0.017
baseline_asr=0.350 defended_asr=0.190 reduction_ci95=[0.120, 0.203]

6.4 Secondary outcomes and costs

Every security improvement has costs: accuracy, latency, refusals of legitimate requests, compute, maintainability. Report them on the same items. A privacy defence that halves leakage but drops macro-F1 by 10 points is a different proposition from one that costs 1 point. Plot the trade-off (e.g., TPR at 1% FPR versus macro-F1) for all conditions. Leo's result is the second kind: near dedup more than halved TPR at 1% FPR, to 4.2%, while macro-F1 fell by less than one point. Report both numbers together, with intervals, so readers can weigh the trade-off themselves.

6.5 Worked example (synthetic Harbourline case)

Leo's main results (synthetic, five seeds, 20,000 records): TPR at 1% FPR under the calibrated attack was 9.8% (original), 7.9% (random removal), 5.1% (exact dedup) and 4.2% (near dedup). The paired difference between near dedup and random removal was 3.7 points (bootstrap 95% CI 2.6–4.9). Macro-F1 fell from 0.912 to 0.905 under near dedup. Data-quality checks found one missing seed-4 run, rerun and logged. An exploratory analysis, labelled as such, showed leakage concentrated in records that had appeared three or more times. The exploratory finding about records seen three or more times is a hypothesis for future work, not a confirmed result.

6.6 Practice activities

  1. Write a data-quality checklist for your study and run it. 2. Produce the primary table with intervals. 3. Plot the security–utility trade-off. For the checklist, include all expected runs, canary checks, duplicates across splits and plausible ranges, and run it before any analysis. For the primary table, give each condition's outcome with an interval and the number of seeds. For the plot, put the privacy metric against macro-F1 for every condition, so a reader can see the trade-off at a glance, as Leo's figure does.

6.7 Errors and limits

Four errors recur, and each biases results or hides costs; check your analysis for all four:

Exercises

Synthesis

Analysis is where careful design pays off: interleaved runs, quality checks, intervals and paired comparisons turn raw logs into a defensible answer, and costs make the answer useful.

Continue studying — chapter 06 of 08

← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →