◇ OpenCyber Academy
Master's · Chapter 5 · English

Chapter 5 — Proposal and pilot

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 5 — Proposal and pilot

Lesson map

Lesson 05 published. Map, example, EX027.

flowchart TD
  N0["Learning outcomes"]
  N1["5.1 The proposal"]
  N2["5.2 The pilot"]
  N3["5.3 Sample size and power"]
  N4["5.4 Revising the plan"]
  N5["5.6 Practice activities"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX027"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["5.1 The proposal"]
  N2["5.2 The pilot"]
  N3["5.3 Sample size and power"]
  N4["5.4 Revising the plan"]
  N5["5.6 Practice activities"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX027"]

[Learning outcomes]
    |
    v
[5.1 The proposal]
    |
    v
[5.2 The pilot]
    |
    v
[5.3 Sample size and power]
    |
    v
[5.4 Revising the plan]
    |
    v
[5.6 Practice activities]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX027]

Learning outcomes

You will (1) write a dissertation proposal with question, background, design, analysis plan, ethics, risks and timeline; (2) run a pilot to test feasibility and estimate variability; (3) compute sample sizes; (4) revise the plan from pilot evidence; and (5) obtain approval with a clear record of decisions.

5.1 The proposal

A master's proposal is typically 3,000–6,000 words. Sections: question and motivation (one paragraph each); background from your evidence matrix; design (Chapter 3) including competing explanations and ablations (Chapter 4); analysis plan with primary and secondary outcomes and the statistics you will use; ethics and data protection; risks (technical, data, schedule) with mitigations; timeline with milestones; and expected contribution stated modestly. The analysis plan is the most neglected part — write it now, before data, so later choices cannot be steered by results. Leo's proposal followed this outline.

5.2 The pilot

A pilot runs the full pipeline at small scale. Goals: confirm the code works end to end, measure run time, measure variability of the primary outcome, check that the attack and metrics behave sensibly (for example, the undefended model is attackable), and find surprises. Pilot results are not hypothesis tests; do not report them as findings, and keep the final test data out of the pilot. Leo's pilot found a floor effect and a generator flaw. Record both in the decision log.

5.3 Sample size and power

For rates, the normal approximation n ≈ z² p(1−p)/h² gives the trials needed for a confidence-interval half-width h. For comparisons between conditions, power analysis asks how many repetitions you need to detect a difference d with probability (power) 0.8 at significance 0.05; for paired designs, the variance of the differences matters. With few seeds (typical in ML), report intervals over seeds and over evaluation items rather than relying on a single significance test. Leo's pilot showed about 0.8 points of variation across seeds, which together with bootstrapping over records made five seeds adequate. Use the variation you actually observe, not a guess.

5.4 Revising the plan

Pilots usually change something: an attack too weak to separate conditions, an evaluation that takes too long, a metric with a floor effect (all conditions near zero). Record each change in a decision log with the reason. Supervisors and examiners respond well to a proposal that shows it learned from its pilot. Leo's log records two changes from his pilot: replacing the floor-limited attack with a calibrated one, and fixing the templates that produced near-identical notes across classes. Each has a date and a reason. Keep the log with the proposal so examiners can trace the plan's evolution.

5.5 Worked example (synthetic Harbourline case)

Leo's pilot used 2,000 records and two seeds. It took 40 minutes per run, so the full plan (20,000 records, four conditions, five seeds) is about 130 hours of compute — feasible on the lab cluster. The simple confidence-threshold attack produced TPR at 1% FPR of 1–2% in all conditions: a floor effect. He replaced it with a reference-model-calibrated attack, which separated conditions. Variability across seeds was about 0.8 points, so five seeds plus bootstrap over records were judged adequate. He also found that two templates generated near-identical notes across classes, introducing label noise; he fixed the generator and recorded the change.

The evasion lab provides an example of a pilot sweep: before choosing one perturbation budget for a main study, sweep several to find where the effect is measurable rather than at floor or ceiling.

"""Lab E: evasion of a toy linear classifier (synthetic data, stdlib only).
Trains logistic regression on two synthetic clusters, then applies a
fast-gradient-sign (FGSM-style) perturbation of size eps to every test point
and reports clean vs adversarial accuracy. Teaching model only."""
import math, random
rng = random.Random(7)
def point(label):
    cx = 1.0 if label else -1.0
    return [rng.gauss(cx, 0.7), rng.gauss(cx, 0.7)], label
train = [point(i % 2) for i in range(400)]
test = [point(i % 2) for i in range(200)]
w, b = [0.0, 0.0], 0.0
sig = lambda z: 1 / (1 + math.exp(-z))
for epoch in range(200):
    gw, gb = [0.0, 0.0], 0.0
    for x, y in train:
        p = sig(w[0]*x[0] + w[1]*x[1] + b)
        gw[0] += (p - y)*x[0]; gw[1] += (p - y)*x[1]; gb += p - y
    w = [w[i] - 0.1*gw[i]/len(train) for i in range(2)]; b -= 0.1*gb/len(train)
def acc(data, eps):
    ok = 0
    for x, y in data:
        # gradient of the loss w.r.t. x is (p - y) * w; FGSM steps along its sign
        p = sig(w[0]*x[0] + w[1]*x[1] + b)
        xa = [x[i] + eps*math.copysign(1, (p - y)*w[i]) for i in range(2)]
        ok += (sig(w[0]*xa[0] + w[1]*xa[1] + b) >= 0.5) == bool(y)
    return ok / len(data)
print(f"weights w=({w[0]:.3f}, {w[1]:.3f}) b={b:.3f}")
for eps in (0.0, 0.25, 0.5, 0.75, 1.0):
    print(f"eps={eps:.2f} accuracy={acc(test, eps):.3f}")

Expected output:

weights w=(1.795, 1.756) b=-0.080
eps=0.00 accuracy=0.985
eps=0.25 accuracy=0.950
eps=0.50 accuracy=0.840
eps=0.75 accuracy=0.660
eps=1.00 accuracy=0.505

5.6 Practice activities

  1. Write your analysis plan with primary and secondary outcomes. 2. Run a pilot and fill a decision log. 3. Compute sample size for your primary outcome. For the analysis plan, name the primary outcome, such as TPR at 1% FPR, and the secondary ones, with the statistics for each. For the pilot, run the whole pipeline at small scale, and log every change with its reason as Leo did. For the sample size, use the formula above with the variation your pilot observed, and say whether your compute budget allows it.

5.7 Errors and limits

Four errors recur, and each weakens the study; check your proposal for all four:

Exercises

Synthesis

The proposal fixes what you will do and how you will analyse it; the pilot shows whether you can. Revisions recorded with reasons turn surprises into evidence of a careful process.

Continue studying — chapter 05 of 08

← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →