◇ OpenCyber Academy
Doctoral research · Chapter 7 · English

Chapter 7 — Thesis synthesis and reproducibility archive

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 7 — Thesis synthesis and reproducibility archive

Lesson map

Lesson 07 published. Map, example, EX039.

flowchart TD
  N0["Learning outcomes"]
  N1["7.1 The argument"]
  N2["7.2 The archive"]
  N3["7.3 Integrity manifest"]
  N4["Lab — archive verification"]
  N5["7.4 Lab result"]
  N6["7.5 What cannot be archived"]
  N7["Writing for examiners"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX039"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["7.1 The argument"]
  N2["7.2 The archive"]
  N3["7.3 Integrity manifest"]
  N4["Lab — archive verification"]
  N5["7.4 Lab result"]
  N6["7.5 What cannot be archived"]
  N7["Writing for examiners"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX039"]

[Learning outcomes]
    |
    v
[7.1 The argument]
    |
    v
[7.2 The archive]
    |
    v
[7.3 Integrity manifest]
    |
    v
[Lab — archive verification]
    |
    v
[7.4 Lab result]
    |
    v
[7.5 What cannot be archived]
    |
    v
[Writing for examiners]
    |
    v
[Worked example] --> [Exercise EX039]

Learning outcomes

You will (1) assemble the thesis argument; (2) package data, code and configuration for reproduction; (3) verify the archive's integrity; (4) document how to rerun.

7.1 The argument

The thesis connects the frontier gap (chapter 2), the defined measures (chapter 3), the replicated baseline (chapter 4), the main result (chapter 5) and the revisions (chapter 6) into one argument, with limits stated in each chapter, not saved for the end.

Stating limits in each chapter means the reader meets them where they arise: the 24-hour log retention in the pilot, the exception workloads in the main study, the workarounds in the critique. A final limitations chapter alone would leave the main result unqualified for most of the thesis, and examiners would have to search for the caveats.

7.2 The archive

The archive contains synthetic and de-identified datasets, analysis scripts, policy configurations used in the study, the label guide, the revision log and a README explaining how to rerun every figure.

The README is tested, not just written: before submission someone other than Amara follows it from a clean machine. The policy configurations matter because they define what admission enforcement actually checked; without them, another researcher could not tell whether a different result came from different clusters or different policies. Every one of these files is listed in the SHA-256 manifest, including config/policies.yaml, the file whose late edit the verification later caught.

7.3 Integrity manifest

A SHA-256 manifest lists every file. Before submission, the archive is verified against it so examiners can trust that results come from the archived files.

The manifest also protects Amara. If an examiner's rerun gave a different figure, the first question would be whether a file had changed, and the manifest answers it in seconds. Verifying before submission found exactly such a change in her real archive: a late edit to a policy file that had not been recorded. That check is what caught the unrecorded edit to the policy file before submission.

Lab — archive verification

"""Build and verify a SHA-256 manifest for a synthetic reproducibility archive (stdlib only)."""
import hashlib
files = {"data/findings.csv": b"id,label\n1,R\n2,N\n", "code/lab_window.py": b"# analysis\n",
         "config/policies.yaml": b"restricted: true\n", "README.md": b"How to rerun the study\n"}
manifest = {p: hashlib.sha256(c).hexdigest() for p, c in files.items()}
files["config/policies.yaml"] = b"restricted: false\n"   # simulate an unrecorded change
bad = [p for p, c in files.items() if hashlib.sha256(c).hexdigest() != manifest[p]]
print(f"files {len(files)}, verified {len(files)-len(bad)}, changed: {', '.join(bad) or 'none'}")

Run with python3 lab_archive.py. Verified output:

files 4, verified 3, changed: config/policies.yaml

7.4 Lab result

Of four synthetic files, three verify; config/policies.yaml has changed since the manifest was made. In the lab the change is simulated, flipping restricted to false; in Amara's real check, a late edit to a policy file had not been recorded. She restores the recorded version and reruns the analysis, confirming no result changes.

The real case is the important one: an unrecorded edit to the policy file meant the archive no longer matched what the study used. Restoring the recorded version and rerunning confirmed the results, and the episode went into the revision log. An examiner later verified the manifest and reran all figures in under an hour.

7.5 What cannot be archived

Raw audit logs from Meridian cannot leave the partner; the archive explains how equivalent data can be generated synthetically using the provided generator.

The generator produces audit records with the same fields, formats and timing patterns as Meridian's exports, so every script runs on it unchanged. It cannot reproduce Meridian's actual results, and the README says so: synthetic runs test the method, while the reported figures come from the partner data that stayed with Meridian. The README states which fields of Meridian's audit logs the generator imitates and that the 30-day retention used in the study clusters is a property of Meridian's setup, not of the generator.

Writing for examiners

Each results chapter opens with the question it answers and closes with what the result does and does not show. Figures carry their data file names, so examiners can trace any number back to the archive. Amara keeps the main text focused on the argument and moves long tables to appendices, with references from the text. Her supervisor reads each chapter for the single sentence a busy examiner would remember, and asks her to make that sentence explicit.

Worked example

An examiner later downloads the archive, verifies the manifest and reruns all figures in under an hour on a laptop, noting this in the examination report as a strength.

Synthetic case. Archive contents are synthetic.

Errors and limits

A manifest proves files are unchanged, not that the analysis is correct.

Exercises

Synthesis

A verified archive lets others rerun the thesis.

Continue studying — chapter 07 of 08

← Chapter 06 (draft) · Course chapter index · Chapter 08 (draft) →