Chapter 7 — Thesis synthesis and reproducibility archive
Lesson map
Lesson 07 published. Map, example, EX039.
flowchart TD
N0["Learning outcomes"]
N1["7.1 Thesis argument"]
N2["7.2 Archive contents"]
N3["Lab — manifest check"]
N4["7.3 Lab result"]
N5["7.4 Fixing the archive"]
N6["7.5 What cannot be shared"]
N7["7.6 Reproducibility check"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX039"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["7.1 Thesis argument"]
N2["7.2 Archive contents"]
N3["Lab — manifest check"]
N4["7.3 Lab result"]
N5["7.4 Fixing the archive"]
N6["7.5 What cannot be shared"]
N7["7.6 Reproducibility check"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX039"]
[Learning outcomes]
|
v
[7.1 Thesis argument]
|
v
[7.2 Archive contents]
|
v
[Lab — manifest check]
|
v
[7.3 Lab result]
|
v
[7.4 Fixing the archive]
|
v
[7.5 What cannot be shared]
|
v
[7.6 Reproducibility check]
|
v
[Worked example] --> [Exercise EX039]
Learning outcomes
You will (1) synthesise studies into a coherent thesis argument; (2) build a reproducibility archive; (3) verify archive integrity with hashes; (4) document what cannot be shared.
7.1 Thesis argument
Chapter by chapter: exposure is measurable (pilot), lower with bots (main study), partly causal (within-project), and limited by tests (interviews). The final chapter links these to SSDF RV.2 and SLSA's emphasis on trustworthy, automated build and update processes. Each step in the argument carries its own limit: the within-project reduction is smaller than the matched difference, and the interviews explain lag only for those who agreed to talk. The link to SSDF RV.2 is framed as evidence for how to meet the practice, not as a measure of compliance.
7.2 Archive contents
Scripts, synthetic replicas, the codebook, the pre-registration, a README describing how to run everything, and a manifest of SHA-256 hashes for every file. The archive also contains the pilot memo and the change log, so readers can see how the protocol evolved. Interview material is limited to the codebook and coded counts, as the ethics approval required. The README lists commands in order and the expected tables, so a colleague can tell at once whether a rerun matched. Script, data and codebook files are all hashed, and the hashes appear in the thesis so a reader can confirm they have the archived versions.
Lab — manifest check
"""Check a synthetic dataset against its archived SHA-256 manifest."""
import hashlib
files = {"exposure.csv": b"project,bot,days\np01,1,3\np02,1,5\n", "codes.csv": b"id,coder_a,coder_b\n1,R,R\n"}
manifest = {"exposure.csv": hashlib.sha256(b"project,bot,days\np01,1,3\np02,1,5\n").hexdigest(),
"codes.csv": hashlib.sha256(b"id,coder_a,coder_b\n1,R,N\n").hexdigest()}
for name, body in files.items():
ok = hashlib.sha256(body).hexdigest() == manifest[name]
print(f"{name:13} {'MATCH' if ok else 'MISMATCH'}")
Run with python3 lab_manifest.py. Verified output:
exposure.csv MATCH
codes.csv MISMATCH
7.3 Lab result
exposure.csv matches its manifest hash; codes.csv does not. The manifest recorded coder B's first label as "N", but the file now says "R", because a late correction was made without updating the manifest. The lab shows the check doing its job: a correct change made silently still fails verification. The fix is not to undo the correction but to record it and regenerate the manifest. Yusuf's real archive is checked the same way before each release, and any mismatch is investigated before a new version is tagged.
7.4 Fixing the archive
Yusuf records the correction in the change log, regenerates the manifest and tags a new archive version. Silent edits break trust even when they are right. The new version has its own tag, so anyone holding the old archive can see that it was superseded and why. The examiner later ran the archive and obtained identical tables, and remarked on the manifest's version history. Keeping both versions accessible means earlier citations still point to something that exists. The log names the coder, the label and the date.
7.5 What cannot be shared
Interview transcripts stay private; the archive holds the codebook and coded counts only. Transcripts could identify maintainers through their projects and wording, even with names removed, so the ethics approval limits release to coded counts. Readers can still check the coding against the codebook and agreement figures. Researchers who want to replicate the qualitative part must collect their own interviews under their own approval. The data access statement explains this choice and points readers to the codebook, the agreement figures and the coded counts that remain available.
7.6 Reproducibility check
A colleague runs the archive on a clean machine from the README alone and reproduces every table. The colleague had not worked on the project and used only the README, which tests the instructions as much as the code. Every table matched, and the run is recorded with its date and archive tag. Any difference would have been fixed before the examiner received the archive. The colleague's notes on unclear steps were used to improve the README before the archive went to the examiner.
Licences
Scripts are released under an open licence and synthetic replicas under CC0, so others may reuse them freely. The README states the licence of each part, the software versions needed and expected running time, which the colleague's clean-machine check confirmed at about twelve minutes. Clear licences matter: reviewers and practitioners hesitate to reuse material whose terms are unclear.
Archive location
The archive is deposited in the university repository with a persistent identifier.
Worked example
The examiner later runs the archive and gets identical tables. She notes the manifest version history as evidence of careful practice.
Synthetic case. Files and hashes are synthetic.
Errors and limits
Hashes prove files are unchanged, not that they are correct.
Exercises
- EX039 — Argument chain. Summarise the thesis argument. Rubric: 3 = measurable; lower with bots; partly causal; limited by tests; 2 = partial; 1 = one; 0 = none.
- EX040 — Archive list. List four archive contents. Rubric: 3 = scripts, replicas, codebook, pre-registration, README, manifest (four); 2 = two; 1 = one; 0 = none.
- EX041 — Lab outcome. What did the manifest lab show? Rubric: 3 = exposure match; codes mismatch from unrecorded correction; 2 = partial; 1 = vague; 0 = none.
- EX042 — Fix. How was the mismatch fixed? Rubric: 3 = change log, regenerate manifest, new version tag; 2 = partial; 1 = vague; 0 = none.
- EX043 — Private data. What stays private? Rubric: 3 = interview transcripts; 2 = partial; 1 = vague; 0 = none.
- EX044 — Hash limit. What do hashes not prove? Rubric: 3 = correctness; 2 = partial; 1 = vague; 0 = none.
Synthesis
A thesis argues across studies and an archive lets others check every step.
Continue studying — chapter 07 of 08
← Chapter 06 (draft) · Course chapter index · Chapter 08 (draft) →