◇ OpenCyber Academy
Master's · Chapter 8 · English

Chapter 8 — Independent review and defense

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 8 — Independent review and defense

Lesson map

Lesson 08 published. Map, example, EX045.

flowchart TD
  N0["Learning outcomes"]
  N1["8.1 Independent review before submission"]
  N2["8.2 The defence presentation"]
  N3["8.3 Answering examiners"]
  N4["8.4 Corrections and dissemination"]
  N5["8.6 Practice activities"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX045"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["8.1 Independent review before submission"]
  N2["8.2 The defence presentation"]
  N3["8.3 Answering examiners"]
  N4["8.4 Corrections and dissemination"]
  N5["8.6 Practice activities"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX045"]

[Learning outcomes]
    |
    v
[8.1 Independent review before submission]
    |
    v
[8.2 The defence presentation]
    |
    v
[8.3 Answering examiners]
    |
    v
[8.4 Corrections and dissemination]
    |
    v
[8.6 Practice activities]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX045]

Learning outcomes

You will (1) commission and use independent review of code and text before submission; (2) prepare a defence presentation; (3) answer examiner questions with evidence and limits; (4) respond to required corrections; and (5) plan dissemination, including a short practitioner summary.

8.1 Independent review before submission

Ask a peer who did not work on the project to (a) reproduce one main table from your artefact appendix, (b) read the method and say what they could not repeat, and (c) read the abstract and say what they think you claim. Each reveals a different problem: hidden dependencies, missing method detail, and overclaiming. A short code review focusing on metric computation and data splits is worth the time; bugs there change conclusions. Leo's peer found a silent skip of long records this way.

8.2 The defence presentation

Twenty to thirty minutes: problem and question; design and why; results with intervals and costs; competing explanations and what ruled them out; limitations; contribution and next steps. One slide should answer the research question in a sentence with a number and an interval. Leo's answer slide gave the paired difference with its interval. Put competing explanations before limitations, since examiners will ask about them. Rehearse with a timer and a peer who asks hard questions, and prepare backup slides for the ablation and the quality checks so you can show evidence when asked.

8.3 Answering examiners

Expect questions on design choices ("why synthetic data?"), threats to validity ("would this hold on real notes?"), statistics ("are five seeds enough?"), alternative explanations ("isn't this just overfitting?") and practice ("what should Harbourline do?"). Answer in three parts: direct answer, evidence, limit. When you do not know, say so and say how you would find out. Leo's answers in the worked example below follow this form: the ablation as evidence, the attack as the limit. Prepare written answers to each expected question and rehearse them with a peer who plays a critical examiner.

8.4 Corrections and dissemination

Examiners often require corrections. Make each change, list it in a correction table with page references, and return it promptly. Dissemination options include a workshop paper, a public repository with the synthetic generator, and a two-page practitioner summary that states the finding, the conditions and the recommended next step — mapped where relevant to frameworks the organisation uses, such as the NIST AI RMF Measure and Manage functions or its ISO/IEC 42001 risk-treatment process. Leo's practitioner summary states deduplication as a low-cost candidate control under the evaluated attack.

8.5 Worked example (synthetic Harbourline case)

A peer reproducing Leo's main table found that the evaluation script silently skipped records longer than 512 tokens, about 3% of non-members. Leo fixed it; TPR at 1% FPR moved by 0.2 points and conclusions held, and he recorded the fix. At the defence an examiner asked: "Isn't this just overfitting?" Leo showed the random-removal ablation and the matched-gap analysis: "Part of the effect is size-related — 1.9 points — but the remaining 3.7 points are specific to duplicates, under this attack." Another asked whether Harbourline should adopt differential privacy instead. Leo answered that DP gives formal guarantees (Abadi et al., arXiv:1607.00133) at a utility cost he did not measure; deduplication is complementary and cheap, and a DP comparison is future work.

The membership-inference lab can be used in the defence to show the core concept in thirty seconds.

"""Lab M: membership inference against an overfit 1-nearest-neighbour 'model'
that returns a confidence score. Synthetic records only; teaching model."""
import random, math
rng = random.Random(5)
def rec(): 
    y = rng.random() < 0.5
    return [rng.gauss(0.6 if y else -0.6, 1.0) for _ in range(4)], int(y)
members = [rec() for _ in range(150)]; nonmembers = [rec() for _ in range(150)]
def confidence(x, y, k):
    # k-NN vote share for the true label, computed over the training set
    d = sorted((sum((a-b)**2 for a, b in zip(x, m)), my) for m, my in members)
    return sum(1 for _, my in d[:k] if my == y)/k
def auc(pos, neg):
    s = sum((p > n) + 0.5*(p == n) for p in pos for n in neg)
    return s/(len(pos)*len(neg))
for k in (1, 5, 25):
    pos = [confidence(x, y, k) for x, y in members]
    neg = [confidence(x, y, k) for x, y in nonmembers]
    best = max(sum(p >= t for p in pos)/len(pos) - sum(n >= t for n in neg)/len(neg)
               for t in sorted(set(pos + neg)))
    print(f"k={k:2d} attack_auc={auc(pos, neg):.3f} best_advantage={best:.3f}")

Expected output:

k= 1 attack_auc=0.600 best_advantage=0.200
k= 5 attack_auc=0.521 best_advantage=0.073
k=25 attack_auc=0.520 best_advantage=0.060

8.6 Practice activities

  1. Arrange a reproduction and abstract-reading review. 2. Build a defence deck with one answer slide. 3. Write the practitioner summary. For the review, ask a peer to reproduce one table and read the abstract, and record what they found. For the deck, include one slide that answers the question with a number and an interval, as Leo's did with the 3.7-point difference. For the summary, state the finding, the conditions and the next step, and keep it to two pages. Keep notes of what changed.

8.7 Errors and limits

Four errors recur, and each weakens the final stage; check your submission for all four:

Exercises

Synthesis

Independent review catches the errors you cannot see; the defence tests whether you can connect every claim to evidence and its limits. A careful practitioner summary carries the result into practice without losing its conditions.

Continue studying — chapter 08 of 08

← Chapter 07 (draft) · Course chapter index · Worked results ↓ · Next course via chapter index →