◇ OpenCyber Academy
Master's · Chapter 3 · English

Chapter 3 — Study design, ethics and reproducibility

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 3 — Study design, ethics and reproducibility

Lesson map

Lesson 03 published. Map, example, EX015.

flowchart TD
  N0["Learning outcomes"]
  N1["3.1 Real or synthetic data"]
  N2["3.2 Ethics approval"]
  N3["3.3 Ground truth"]
  N4["3.4 Measures"]
  N5["3.5 Controls"]
  N6["3.6 Reproducibility plan"]
  N7["3.7 Tool validation"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX015"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["3.1 Real or synthetic data"]
  N2["3.2 Ethics approval"]
  N3["3.3 Ground truth"]
  N4["3.4 Measures"]
  N5["3.5 Controls"]
  N6["3.6 Reproducibility plan"]
  N7["3.7 Tool validation"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX015"]

[Learning outcomes]
    |
    v
[3.1 Real or synthetic data]
    |
    v
[3.2 Ethics approval]
    |
    v
[3.3 Ground truth]
    |
    v
[3.4 Measures]
    |
    v
[3.5 Controls]
    |
    v
[3.6 Reproducibility plan]
    |
    v
[3.7 Tool validation]
    |
    v
[Worked example] --> [Exercise EX015]

Learning outcomes

You will (1) choose between real and synthetic evidence; (2) obtain ethical and legal approval; (3) design measurements of reduction and misses; (4) plan for reproducibility.

3.1 Real or synthetic data

Real casework images hold personal data and are restricted by law and by court orders. Garfinkel et al. (2009) argued for standardised, shareable forensic corpora so that results can be compared. Tomás uses synthetic laptop images he builds: a clean OS install, standard applications and planted user documents, so the ground truth is known. Building images himself means every file's origin and label are known before filtering, so misses can be counted exactly. The cost is realism: synthetic images lack the clutter of real laptops, which he records as a limitation.

3.2 Ethics approval

Even synthetic work needs ethics review if it involves people (an examiner agreement study does). He applies to the university committee, describing consent for examiner participants and confirming no casework data leave the police unit. His application explains that examiners will classify events from synthetic images only, that their results will be anonymised, and that they can withdraw. It confirms that his police role gives him no access to the study's data beyond what any participant would see. The committee approves the application before any examiner is recruited, and he keeps the approval letter in the project archive.

3.3 Ground truth

Every planted file is labelled "relevant" or "not relevant" before filtering. A miss is a relevant file set aside as known. Labelling before filtering prevents hindsight from changing what counts as relevant. Tomás records each label with the reason, such as a user document related to the synthetic scenario or a legitimate tool planted with illegitimate use. Designed misses are labelled relevant from the start. When a check prompted by his committee found that one planted manual was identical to an NSRL entry, he relabelled it as a designed miss with a documented reason rather than deleting it. The labels are archived with each image's seed, so anyone rebuilding an image also gets the ground truth used to count its misses.

3.4 Measures

Reduction = known files ÷ all files. Miss rate = relevant files set aside ÷ relevant files. Both are reported with counts. Reporting counts as well as percentages lets readers see how many files a figure is based on: a miss rate from a handful of relevant files means less than one from hundreds. The two measures pull in opposite directions, since a filter that sets aside more files raises reduction but may also raise misses. Tomás reports both for each image so that neither is presented alone. His results tables show the file counts behind every reduction and miss percentage, image by image.

3.5 Controls

He fixes the reference set version and hashing tool version, and records them, because a newer reference set changes results. Freezing these versions turns the reference set into a controlled condition rather than a moving one. Without it, a release added during the study could change which files count as known, and differences between images could reflect the date they were processed. Tomás records the version and hash tool in every log entry. His pilot later showed how strongly reduction depends on how current the reference set is, which made the control even more important.

3.6 Reproducibility plan

Scripts, image build recipes, seed values and the reference set version are archived; every result traces to a script run. Seed values mean the image builder produces the same files each time, so another researcher can recreate the images rather than trusting his description. Tomás tests the plan during the pilot by rebuilding an image from the archive and checking that file hashes match. The archive is released with the thesis, apart from the examiners' raw responses, which stay anonymised under the ethics approval.

3.7 Tool validation

NIST's Computer Forensics Tool Testing programme publishes test methods and reports; Tomás checks his hashing tool against known test vectors before use. A hashing error would make files appear unknown or, worse, known when they are not, so tool validation protects both measures. Tomás records which test vectors he used, the tool version and the results. When the pilot produced an unexpectedly low reduction on one image, he could rule out tool failure partly because the tool had passed these checks, and a second tool gave identical hashes. The validation record is part of the archive, so examiners can check the tool version against the test vectors he used.

Data management plan

The plan lists where images and scripts are stored, who can access them, and how long they are kept. Because the images are synthetic, they can be published after the study.

Worked example

His committee asks whether planted "relevant" files could accidentally equal published files. He checks: one planted PDF was a vendor manual identical to an NSRL entry. He relabels it carefully as a designed miss, documenting why, rather than deleting it.

Synthetic case. The committee exchange is invented.

Errors and limits

Synthetic images can lack the clutter of real laptops; reduction measured on them may overstate real-world effect.

Exercises

Synthesis

Known ground truth, fixed versions, ethics approval and archived scripts turn a triage idea into a defensible study.

Continue studying — chapter 03 of 08

← Chapter 02 (draft) · Course chapter index · Chapter 04 (draft) →