◇ OpenCyber Academy
Doctoral research · Chapter 7 · English

Chapter 7 — Thesis synthesis and reproducibility archive

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 7 — Thesis synthesis and reproducibility archive

Lesson map

Lesson 07 published. Map, example, EX043.

flowchart TD
  N0["Learning outcomes"]
  N1["7.1 One claim per chapter"]
  N2["7.2 What the archive contains"]
  N3["7.3 Environment"]
  N4["7.4 Data-availability statement"]
  N5["7.5 Machine-readable outputs"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX043"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["7.1 One claim per chapter"]
  N2["7.2 What the archive contains"]
  N3["7.3 Environment"]
  N4["7.4 Data-availability statement"]
  N5["7.5 Machine-readable outputs"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX043"]

[Learning outcomes]
    |
    v
[7.1 One claim per chapter]
    |
    v
[7.2 What the archive contains]
    |
    v
[7.3 Environment]
    |
    v
[7.4 Data-availability statement]
    |
    v
[7.5 Machine-readable outputs]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX043]

Learning outcomes

You will (1) structure a thesis so each chapter carries one claim; (2) build a reproducibility archive that excludes restricted data lawfully; (3) document software environment and random seeds; (4) write a data-availability statement that is honest about restrictions.

7.1 One claim per chapter

Readers should be able to state each chapter's claim in a sentence: integrity design; gap; proposal; baseline replication; main calibration result; robustness under critique. If a chapter has three unrelated claims, split it; if it has none, cut it. Tomas applied the test to each of his six chapters: a chapter whose claim could not be stated in a sentence was split or cut, and the test caught a draft chapter mixing the gap with the proposal. For the calibration chapter, the sentence has to name the measure and the corpus scope, as EX043 asks.

7.2 What the archive contains

Code, codebook versions, synthetic test data, aggregate result tables, the pre-registration, and a script that rebuilds every figure. Restricted inputs — TLP:AMBER+STRICT reports, licensed vendor text — are replaced by a manifest of identifiers and SHA-256 hashes so that someone with lawful access can verify they hold the same files.

Tomas's archive holds 14 scripts, 3 codebook versions, a manifest of 1,200 report hashes and synthetic samples that exercise every code path. A colleague with no lab access rebuilt every figure in under an hour, which is the test that matters.

7.3 Environment

Record the Python version and confirm the analysis uses only the standard library, or pin every dependency. Record any random seed used for sampling. "It ran on my laptop" is not reproducibility.

Tomas's labs use only the standard library, which removes most dependency drift; the archive still records the Python version used. The main sample's selection uses a recorded seed, so the 340 judgments can be drawn again from the manifest by anyone with lawful access to the reports. The archive also records the operating system the figures were built on, because the colleague's one-hour rebuild was done on a different machine.

7.4 Data-availability statement

State what is open, what is restricted and why, and how an authorised researcher can request access. Do not say "available on request" if requests will in practice be refused.

For Tomas the restricted part is the 200 TLP:AMBER+STRICT reports and licensed vendor text. His statement names the sharing community and the licence holder as the routes for access, and the hash manifest lets an authorised researcher confirm they hold the same files. It also says what an authorised researcher cannot get: the reports themselves never leave the lab, so access means working on site under the same TLP:AMBER+STRICT rules that bound Tomas.

7.5 Machine-readable outputs

Where the thesis produces reusable structures — for example a mapping of estimative phrases to probability ranges — publish them as plain JSON or CSV with a schema. If the work creates threat objects, STIX 2.1 and TAXII 2.1 provide standard exchange formats, but a methods thesis rarely needs them.

Tomas's phrase-to-probability mapping is the obvious candidate: a CSV with one row per estimative phrase, its numeric range and the codebook version, so that others can apply the same mapping to their own reports.

Worked example

A colleague with no lab access takes only the archive, runs the rebuild script against the synthetic samples and the aggregate tables, and compares each figure with the thesis draft; every figure matches, and the whole rebuild takes under an hour.

Synthetic case. The archive contents and rebuild test are invented.

Errors and limits

Hash manifests prove identity of files, not lawful access. Archive hosting must respect the same restrictions as the data.

Exercises

Synthesis

One claim per chapter, an archive that rebuilds everything lawful, honest availability, and restricted data represented by hashes alone.

Continue studying — chapter 07 of 08

← Chapter 06 (draft) · Course chapter index · Chapter 08 (draft) →