Chapter 7 — Thesis synthesis and reproducibility archive
Lesson map
Lesson 07 published. Map, example, EX039.
flowchart TD
N0["Learning outcomes"]
N1["7.1 Claims from evidence"]
N2["7.2 Limitations"]
N3["7.3 Reproducibility archive"]
N4["7.4 Secrets vs science"]
N5["7.5 Thesis structure"]
N6["7.5b Reading guide for practitioners"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX039"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["7.1 Claims from evidence"]
N2["7.2 Limitations"]
N3["7.3 Reproducibility archive"]
N4["7.4 Secrets vs science"]
N5["7.5 Thesis structure"]
N6["7.5b Reading guide for practitioners"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX039"]
[Learning outcomes]
|
v
[7.1 Claims from evidence]
|
v
[7.2 Limitations]
|
v
[7.3 Reproducibility archive]
|
v
[7.4 Secrets vs science]
|
v
[7.5 Thesis structure]
|
v
[7.5b Reading guide for practitioners]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX039]
Learning outcomes
You will (1) synthesise claims that follow from evidence; (2) state limitations without apology theatre; (3) build a reproducibility archive others can run; (4) separate scientific artefacts from operational secrets; and (5) prepare the thesis structure.
7.1 Claims from evidence
Each claim cites a measure, a sample and conditions. Example: "On NHUSTL's synthetic ranges, enabling machine-checkable scope predicates reduced out-of-scope probes from 0.17/hour to 0.03/hour (20 engagements per arm)." Claims that need exploit detail are rewritten as measurement claims.
Elena's sentence names the setting, the rates and the sample. A claim such as "the lab is secure" would need exploit detail or a universal score; she rewrites every such draft as a measurement claim with conditions. Every claim in the results chapter should pass this test.
7.2 Limitations
List: synthetic topology; student operators; single checker implementation; English-only ROE; no live enterprise network. Limitations are not failures — they bound the claim.
Elena places the list before the discussion. Each line narrows where the 0.17 to 0.03 drop can be claimed: not yet on live enterprise networks, not yet with professional operators, not yet with other checker designs. Limitations are part of the contribution because they stop unsafe generalisation into live enterprise programmes. Each line is a bound on the claim, not an apology.
7.3 Reproducibility archive
Include: anonymised logs schema; checker source; scoring scripts; pre-registration; analysis notebooks that regenerate tables from raw data; environment pins. A peer should reproduce the primary table on a clean machine.
"""Verify integrity of a synthetic evidence package."""
import hashlib, json
files = {
"roe.pdf": b"SYNTHETIC-ROE-v1",
"scope.csv": b"host,port\n10.20.1.10,443\n",
"findings.json": b'[{"id":"SYN-001"}]',
}
expected = {n: hashlib.sha256(b).hexdigest()[:16] for n,b in files.items()}
# Simulate tampering of findings.json
current = dict(expected)
current["findings.json"] = hashlib.sha256(b'[{"id":"SYN-001"},{"id":"TAMPERED"}]').hexdigest()[:16]
print(json.dumps(expected, indent=1))
for n in sorted(expected):
status = "OK" if expected[n] == current[n] else "MISMATCH"
print(f"{n:16} {status}")
Expected output:
{
"roe.pdf": "18d74983d403e7db",
"scope.csv": "01cc4815eddf3de6",
"findings.json": "7f5f178ab43ef00f"
}
findings.json MISMATCH
roe.pdf OK
scope.csv OK
Elena's archive regenerates the primary table — prose 0.17/hour against checker 0.03/hour, twenty engagements per arm — from the anonymised logs. Environment pins record the Python and checker versions. A peer who cannot rebuild the table on a clean machine has found a gap. Hash the release so examiners can detect accidental edits after pinning.
7.4 Secrets vs science
Keep out of the archive: real credentials, client names, weaponised procedures. Keep in: synthetic topologies, measures, code for scoping and scoring. Hash-pin the archive; the lab shows how a tampered findings file is detected.
Elena's private appendix holds authorization templates without signatures. The hash lab shows a tampered findings.json as MISMATCH while roe.pdf and scope.csv stay OK; she investigates, restores from a trusted copy and re-hashes before submission. Hash-pinning is what makes the archive trustworthy after it leaves your machine.
7.5 Thesis structure
Introduction; background (NIST 800-115, PTES, ATT&CK, TIBER-EU/CBEST, measurement critiques); related work; methods; results; discussion; ethics; archive guide; conclusion.
Elena's abstract states the question, measure and primary result in under 250 words, so a reader finds the claim before the archive guide. Each chapter maps to one stage of the doctorate: methods hold the pre-registration and ethics; results hold the rates; discussion and limitations bound the claim; and the archive guide tells a peer how to rebuild the primary table. Keep the order so examiners can find each piece quickly.
7.5b Reading guide for practitioners
Include a two-page reading guide at the front of the archive: how to install, how to replay the primary table, and which files are deliberately absent (and why). Busy assessors will not reverse-engineer your repository layout.
Elena's guide names the missing files — signed authorizations, any real vulnerability details — and the dual-use reason they are absent. Install and replay steps are numbered, with expected output matching the primary table. A colleague outside the project tests the guide on a clean machine before submission and reports any step that fails.
7.6 Worked example (synthetic NHUSTL case)
Elena's archive is a public repository with synthetic data and the checker. A private appendix holds institutional authorization templates without signatures. She verifies archive integrity with hashes before submission. Her synthesis chapter rejects the sentence "the lab is secure" in favour of the measured rate reduction under stated conditions.
The public repository is what examiners and later researchers can run. The private appendix stays inside the university. Hash verification before submission catches accidental edits that would make a rebuilt table disagree with the results chapter.
7.7 Errors and limits
- Archives that do not run.
- Claims broader than conditions.
- Mixing client secrets into public artefacts.
Each error weakens the thesis. Archives that do not run cannot support the reproducibility claim. Claims broader than the conditions invite readers to apply results where they were never tested. Client secrets in public artefacts turn research into an exposure. Check the archive and synthesis chapter for all three before submission. Ask a reviewer to check them too. Elena's reviewer replays the primary table from the public archive and searches it for client names before she submits.
Exercises
- EX039 — Claim. Write one evidence-bound claim. Rubric: 3 = measure, sample, conditions; 2 = two; 1 = slogan; 0 = none.
- EX040 — Limits. List five limitations. Rubric: 3 = five apt; 2 = four; 1 = two; 0 = none.
- EX041 — Archive. List six archive contents. Rubric: 3 = six; 2 = four; 1 = two; 0 = none.
- EX042 — Hash lab. What does MISMATCH on findings.json imply? Rubric: 3 = tamper/change since pin; block trust; investigate; 2 = two; 1 = one; 0 = wrong.
- EX043 — Secrets. Classify four items as public archive vs private. Rubric: 3 = four correct; 2 = three; 1 = one; 0 = none.
- EX044 — Structure. Outline eight thesis chapters. Rubric: 3 = coherent order; 2 = most; 1 = few; 0 = none.
Synthesis
Synthesis ties claims to evidence; the archive lets others check you; secrets stay out of the science pack.
Continue studying — chapter 07 of 08
← Chapter 06 (draft) · Course chapter index · Chapter 08 (draft) →