◇ OpenCyber Academy
Master's · Chapter 5 · English

Chapter 5 — Proposal and pilot

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 5 — Proposal and pilot

Lesson map

Lesson 05 published. Map, example, EX027.

flowchart TD
  N0["Learning outcomes"]
  N1["5.1 Proposal sections"]
  N2["5.2 Timeline"]
  N3["5.3 Risks"]
  N4["Lab — known-file filtering"]
  N5["5.4 Reading the result"]
  N6["5.5 Why a pilot"]
  N7["5.6 Revisions"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX027"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["5.1 Proposal sections"]
  N2["5.2 Timeline"]
  N3["5.3 Risks"]
  N4["Lab — known-file filtering"]
  N5["5.4 Reading the result"]
  N6["5.5 Why a pilot"]
  N7["5.6 Revisions"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX027"]

[Learning outcomes]
    |
    v
[5.1 Proposal sections]
    |
    v
[5.2 Timeline]
    |
    v
[5.3 Risks]
    |
    v
[Lab — known-file filtering]
    |
    v
[5.4 Reading the result]
    |
    v
[5.5 Why a pilot]
    |
    v
[5.6 Revisions]
    |
    v
[Worked example] --> [Exercise EX027]

Learning outcomes

You will (1) write a master's proposal for a forensic study; (2) run a pilot; (3) compute reduction on synthetic data; (4) revise the plan from pilot findings.

5.1 Proposal sections

Question; background; method (images, reference set, tools, measures); ethics and legal; risks; timeline; outputs. Tomás keeps it to twelve pages. Twelve pages force him to state the method clearly enough for the committee to judge feasibility, without the detail that belongs in the thesis. The ethics and legal part explains why images are synthetic and how examiner participants are protected. Outputs include the thesis, the released image builder and a practitioner summary. Dr Osei reviews a full draft before it goes to the committee.

5.2 Timeline

Months 1–4 literature and image building; 5–8 pilot and main runs; 9–12 agreement study; 13–18 analysis and writing; 19–24 review and defence. Placing the agreement study after the main runs means examiners classify events from the final reduced sets rather than from pilot images. Writing starts in month 13, but Tomás drafts the literature review and method chapters as he goes, so the last months are spent on results and revision. Each phase ends with a short report to Dr Osei, who checks progress against the plan. The timeline is part of the proposal the committee approves, so a slipped phase has to be explained against it.

5.3 Risks

Reference set update mid-study (mitigation: freeze a version); police unit access changes (mitigation: all data synthetic); participant recruitment for the agreement study (mitigation: recruit from two units). Each risk is paired with a mitigation that can be applied before the risk occurs. Freezing the reference set version removes the first risk entirely. Using only synthetic images means a change in his police unit's access rules cannot stop the study. Recruiting from two units makes it more likely that enough examiners take part in the agreement study. Tomás reviews the risk list with Dr Osei at each phase.

Lab — known-file filtering

"""Known-file filtering on a synthetic triage set (stdlib only)."""
import hashlib
known = {hashlib.sha256(f"os-file-{i}".encode()).hexdigest() for i in range(800)}
files = [f"os-file-{i}" for i in range(0, 800, 2)] + [f"user-doc-{i}" for i in range(100)]
hits = sum(hashlib.sha256(f.encode()).hexdigest() in known for f in files)
print(f"files={len(files)} known={hits} left_for_review={len(files)-hits}")
print(f"reduction={hits/len(files):.1%}")

Run with python3 lab_nsrl.py. Verified output:

files=500 known=400 left_for_review=100
reduction=80.0%

5.4 Reading the result

Of 500 synthetic files, 400 match the synthetic reference set and 100 remain for review: an 80.0% reduction. All 100 remaining files are user documents, and none of the user documents were set aside, so in this pilot the miss count is zero. The zero miss count shows that the scripts worked, not that filtering is safe: the pilot planted no designed misses, so there was nothing for filtering to hide. Tomás states this beside the pilot figures, so no reader mistakes them for a test of the hypothesis. The main run adds designed misses.

5.5 Why a pilot

The pilot tests scripts, timing and measures, not the hypothesis. It found the hashing step took four minutes per image, allowing the main study budget to be set. Because the pilot tests procedure, its results are not combined with the main run's. Its main lesson for the budget is time: at four minutes of hashing per image, thirty images need two hours of hashing, leaving most of the time for building images and checking labels. The pilot also found the low-reduction image that led to a new variable.

5.6 Revisions

The pilot showed the image builder placed no third-party software; Tomás adds common applications so the reference set's coverage of applications is tested too. Without third-party applications, the images contained only operating system files and user documents, so the study could not show how well the reference set covers common software. Adding applications also allows designed misses involving legitimate tools to be planted realistically. The revision is recorded in the protocol before the main run, with the pilot finding that prompted it.

5.7 Supervisor review

Dr Amara Osei asks for a stopping rule: the main run uses 30 images, fixed in advance, so results are not tuned by adding images until numbers look good. Fixing the number in advance protects the result from a common temptation: running more images until an effect appears or disappears. Thirty images is enough to show how reduction changes with the gap to the reference release, and it fits within the time budget set by the pilot. Tomás records the stopping rule in the protocol. If an image fails to build, it is reported as missing rather than replaced.

Budget

Storage for thirty images, a workstation and examiner time for the agreement study are costed. The police unit contributes no data, only optional advice from staff in their own time.

Worked example

Tomás writes in the proposal: "Primary measure: reduction with counts. Secondary: miss rate against planted ground truth. Pilot: 500 files, 80.0% reduction, 0 misses — expected, as the pilot planted no designed misses."

Synthetic case. The pilot data are generated by the lab script.

Errors and limits

A zero-miss pilot says nothing about real miss rates; designed miss cases must be added.

Exercises

Synthesis

A short proposal, a working pilot, and a pre-fixed stopping rule prepare the main study.

Continue studying — chapter 05 of 08

← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →