◇ OpenCyber Academy
Master's · Chapter 5 · English

Chapter 5 — Proposal and pilot

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 5 — Proposal and pilot

Lesson map

Lesson 05 published. Map, example, EX027.

flowchart TD
  N0["Learning outcomes"]
  N1["5.1 The proposal"]
  N2["5.2 Why label a sample"]
  N3["Lab — stratified sample"]
  N4["5.3 Reading the result"]
  N5["5.4 The pilot"]
  N6["5.5 Logistics found"]
  N7["5.6 Proposal approval"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX027"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["5.1 The proposal"]
  N2["5.2 Why label a sample"]
  N3["Lab — stratified sample"]
  N4["5.3 Reading the result"]
  N5["5.4 The pilot"]
  N6["5.5 Logistics found"]
  N7["5.6 Proposal approval"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX027"]

[Learning outcomes]
    |
    v
[5.1 The proposal]
    |
    v
[5.2 Why label a sample]
    |
    v
[Lab — stratified sample]
    |
    v
[5.3 Reading the result]
    |
    v
[5.4 The pilot]
    |
    v
[5.5 Logistics found]
    |
    v
[5.6 Proposal approval]
    |
    v
[Worked example] --> [Exercise EX027]

Learning outcomes

You will (1) write a dissertation proposal; (2) plan a labelling sample efficiently; (3) pilot data collection and labelling; (4) adjust the plan from the pilot.

5.1 The proposal

Sections: background and gap; question and hypotheses; design; data and ethics; analysis; validity threats; timeline; risks. Dr. Solberg asks for one page per section at most.

The page limit forces each part to commit. The design page names the twenty rules and the matching criteria; the analysis page names difference-in-differences and the bootstrap; the risks page names what would stop the study, such as the purple team being unavailable. Dr. Solberg can read the whole proposal in one sitting and see every decision.

5.2 Why label a sample

Volume alone cannot show whether suppressed alerts were benign. Tariq will manually label a sample of alerts as benign trigger, false positive or true positive, to estimate how much of the reduction removed only benign noise. Labelling all 4,000 alerts is impossible; a stratified sample by severity gives precise estimates where it matters.

The three labels answer different questions. A benign trigger is noise the SOC can safely remove; a false positive points to a rule that needs fixing; a true positive removed by an allow-list is a suppressed attack. Stratifying by severity ensures the high-severity alerts, where a suppressed attack would matter most, are not represented by a handful of cases.

Lab — stratified sample

"""Stratified sample of synthetic alerts for manual labelling."""
import random
random.seed(7)
population = {"high": 120, "medium": 900, "low": 2980}
budget = 200
total = sum(population.values())
alloc = {k: max(20, round(budget * v / total)) for k, v in population.items()}
over = sum(alloc.values()) - budget
alloc["low"] -= over
for k, v in population.items():
    ids = random.sample(range(v), alloc[k])
    print(f"{k:6} population={v:5} sampled={alloc[k]:3} first ids={sorted(ids)[:3]}")
print("total sampled", sum(alloc.values()))

Run with python3 lab_strata.py. Verified output:

high   population=  120 sampled= 20 first ids=[4, 6, 7]
medium population=  900 sampled= 45 first ids=[47, 50, 60]
low    population= 2980 sampled=135 first ids=[16, 49, 92]
total sampled 200

5.3 Reading the result

With a budget of 200, proportional allocation would give the high-severity stratum only 6 alerts, too few to estimate anything. A minimum of 20 is set, giving high 20, medium 45, low 135. A fixed seed (7) makes the draw reproducible; estimates for the whole population are reweighted by stratum size.

Raising the high stratum to 20 takes alerts from the low stratum, which drops from 149 to 135 to keep the total at 200. That is the right trade: low-severity alerts are numerous and similar, so 135 still estimate their composition well, while 6 high-severity alerts would have made any statement about that stratum meaningless.

5.4 The pilot

Tariq and a SOC colleague label 30 alerts independently. They disagree on 6, mostly "benign trigger" versus "false positive". The definitions are sharpened: a benign trigger is the intended behaviour occurring legitimately; a false positive is the rule matching behaviour it was not meant to catch.

Six disagreements in 30 alerts is a 20% disagreement rate, far too high for the main labelling. Sharpening the definitions before the main sample, rather than after, means the labelling of 200 alerts uses one stable rule. The pilot also shows where the definitions were fuzzy, which goes into the limitations as a possible source of shared bias.

5.5 Logistics found

Exporting alerts with pseudonymised fields took two hours by hand; Tariq scripts it. The SOC's ticket system truncates notes; he uses the SIEM export instead.

Both fixes matter for the main study. A two-hour manual export each week would invite shortcuts or errors over eight weeks; a script runs identically every time and can be archived. Truncated ticket notes would have lost the analysts' reasons for closing alerts, which the labellers need to decide between benign trigger and false positive.

5.6 Proposal approval

Solberg and the SOC lead Marcus Bell sign the proposal; the CISO signs the purple-team annex.

Three signatures reflect three kinds of authority: Solberg for the academic design, Bell for the SOC's data and staff time, and the CISO for authorised testing on Pelham's systems. The purple-team annex sets the scope, including the exclusion of operational technology networks, so the exercise lead can refuse anything outside it. The CISO's annex is the document the exercise lead carries during the injections, so that the scope can be checked on the day without calling anyone.

Worked example

Proposal timeline: "Weeks 1–4 baseline and labelling of baseline sample; week 5 allow-lists deployed after peer review; weeks 5–8 follow-up; week 9 purple-team summary; weeks 10–14 analysis and writing."

Synthetic case. Sample figures come from the lab script.

Errors and limits

Minimum stratum sizes improve small-stratum estimates but require reweighting; forgetting this biases totals.

Exercises

Synthesis

A proposal sets the plan; a pilot tests definitions and logistics; sampling makes manual labelling feasible and reproducible.

Continue studying — chapter 05 of 08

← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →