Chapter 8 — Thesis synthesis and reproducibility archive

Chapter 8 — Thesis synthesis and reproducibility archive

Lesson map

Lesson 08 published. Map, example, EX050.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["8.1 The thesis as an argument"]
  N2["8.2 Integrating failures and narrowed claims"]
  N3["8.3 The reproducibility archive"]
  N4["8.4 Data governance after submission"]
  N5["8.5 Limitations and future work"]
  N6["8.6 Preparing the defense"]
  N7["8.8 Practical activity: archive audit"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX050"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["8.1 The thesis as an argument"]
  N2["8.2 Integrating failures and narrowed claims"]
  N3["8.3 The reproducibility archive"]
  N4["8.4 Data governance after submission"]
  N5["8.5 Limitations and future work"]
  N6["8.6 Preparing the defense"]
  N7["8.8 Practical activity: archive audit"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX050"]

[Learning outcomes and connection]
    |
    v
[8.1 The thesis as an argument]
    |
    v
[8.2 Integrating failures and narrowed claims]
    |
    v
[8.3 The reproducibility archive]
    |
    v
[8.4 Data governance after submission]
    |
    v
[8.5 Limitations and future work]
    |
    v
[8.6 Preparing the defense]
    |
    v
[8.8 Practical activity: archive audit]
    |
    v
[Worked example] --> [Exercise EX050]

Learning outcomes and connection

By the end of this chapter the learner will be able to:

1. Structure a thesis as an argument whose main claim is traceable to evidence, qualifications and answered objections.
2. Assemble a reproducibility archive with documented provenance, integrity manifests, permissions and release tiers.
3. Write a data-governance plan covering access, retention, destruction and correction after the thesis is submitted.
4. State limitations and future work precisely enough to guide the next researcher.
5. Prepare a defense that anticipates the strongest objections.

This chapter integrates the whole book. Chapter 1's integrity and reproducibility standards become the archive's requirements; Chapters 2–7 supply the content of the argument. The learner leaves with a thesis whose claims are exactly as strong as its evidence.

8.1 The thesis as an argument

A doctoral thesis is not a report of everything done; it is an argument that a contribution is warranted. A classic model of practical argument, associated with the philosopher Stephen Toulmin, distinguishes a claim, the grounds (evidence) offered for it, the warrant connecting grounds to claim, backing for the warrant, a qualifier expressing strength, and rebuttals (conditions under which the claim would not hold). The model is useful here because it maps directly onto this book's vocabulary:

In plain terms: The thesis is one argument: a claim, the evidence for it, the reason the evidence supports it, and the limits — each traceable to a chapter.

The thesis chapters should follow the argument, not the chronology of the project. A reader should be able to find, for any sentence in the conclusion, the grounds, warrant and qualifiers that support it.

8.2 Integrating failures and narrowed claims

Doctoral work rarely ends with the claim it started with. The Handoff Study began with "SHA improves comprehension and projection" and ended with "SHA reduces gap-misread errors where telemetry is sparse." That trajectory is a strength when it is documented: the thesis should show the original claim, the evidence and critique that narrowed it, and the final claim. Hiding the trajectory invites the suspicion that the final claim was chosen after the fact (Chapter 1's HARKing). A claim-evolution table in the introduction or conclusion makes the trajectory explicit:

In plain terms: The claim narrowed at each stage, and every narrowing has a cited cause; that trail is what makes the final claim credible.

8.3 The reproducibility archive

The archive is the thesis's evidence made inspectable. It implements the tiered plan from Chapter 1 (§1.4.5) and should be designed to answer a reviewer's question: "Can I verify each number, and if not, why not?"

8.3.1 Structure

handoff-study-archive/
  README.md                 # what is here, how to run, what is NOT here and why
  CITATION.cff              # how to cite the archive (optional but useful)
  LICENSES.md               # licenses per component (text, code, data)
  PERMISSIONS.md            # permissions register: what each partner/participant allowed
  MANIFEST.sha256           # integrity manifest for every file
  protocol/                 # protocol versions, preregistrations, deviation log, pilot report
  instruments/              # SHA template, scenarios, codebook, rater training, consent forms (blank)
  data/
    synthetic/              # schema-matched synthetic data for running the pipeline (T1)
    aggregate/              # released aggregates with suppression notes (T2)
    restricted/README.md    # description of restricted data and access procedure (T3/T4); no data
  code/                     # analysis scripts, environment lock file, run log
  results/                  # outputs regenerated by code, with hashes
  review/                   # reviewer comments and responses, red-team notes, reproduction reports
  CHANGELOG.md              # corrections after release

8.3.2 Provenance and FAIR metadata

Every dataset needs a provenance record: source, permission, collection dates, transformations (with scripts), version and hash. Even restricted data can follow the FAIR principles at the metadata level: the restricted dataset's description can be findable (indexed in a repository with a persistent identifier), its access procedure accessible (documented), its schema interoperable (standard formats and a data dictionary) and its terms reusable (clear conditions). FAIR does not require that the data themselves be open.

8.3.3 Licenses and rights

Use explicit licenses per component. In this repository, original maintainer-owned teaching text is dedicated under CC0 1.0; a thesis archive may choose differently. Data released under a data-use agreement must not be relicensed openly. Cited works retain their own rights. Consent forms determine what participant data can ever be released; the permissions register records those limits so that later archive maintainers do not exceed them.

8.3.4 Integrity

A cryptographic hash manifest lets anyone verify that files have not changed since release. The technical book's record-integrity chapter covers the operational use of digests; in an archive, the manifest is regenerated at each release and recorded in the change log.

8.4 Data governance after submission

Governance continues after the defense. The plan should specify:

• Custodian. Who is responsible for restricted data after the candidate leaves the program.
• Access procedure. How a qualified researcher requests restricted data or enclave access; who decides; what agreements are required.
• Retention and destruction. How long restricted data are kept, as permitted by the data-use agreement and consent, and how destruction is verified and recorded.
• Re-identification response. What happens if someone reports that released data can be re-identified: withdraw the release, notify affected parties as required, and record the event.
• Corrections. How errors are reported and corrected (Chapter 7 §7.8), with change-log entries and, where applicable, notices through the publication venue.

8.5 Limitations and future work

A limitations section is not a list of apologies. Each limitation should state what it limits (which claim), in what direction it could bias results (if known), and what would resolve it. For the Handoff Study:

In plain terms: Each limitation is tied to the claim it weakens and to the study that could resolve it; this is the honest source of future work.

Future work follows from the limitations and the untested parts of the mechanism. It should be specific enough that another researcher could begin: name the question, the design and the first falsifying observation.

8.6 Preparing the defense

A defense tests whether the candidate understands the limits of their own work. Preparation:

1. Objection inventory. Collect every serious objection from reviewers, red-team readers and the pre-mortem (Chapter 7). For each, prepare a short answer with evidence and a slide or reference location.
2. Strongest-objection rehearsal. Ask a colleague to argue the strongest case against the main claim; answer without defensiveness.
3. Archive walkthrough. Be ready to show, live, how a reported number is regenerated from the archive.
4. Scope discipline. Practice answering "Does SHA improve incident response?" with the claim as actually supported, not the claim one wishes were supported.

8.7 Worked example: the study protocols, synthetic datasets, analysis code and reviewer exchanges

Synthetic. The candidate assembles the Handoff Study archive. The script below builds a miniature version in a temporary directory, writes an integrity manifest, verifies it, then simulates an unrecorded edit to the deviation log and shows that verification detects it. It writes nothing outside the temporary directory, which is deleted when the script ends.

"""Chapter 8 offline lab: build and verify a hash manifest for a synthetic thesis archive.
Writes only inside a temporary directory that is deleted afterwards."""
import hashlib, tempfile
from pathlib import Path

FILES = {
    "protocol/protocol_v1.2.md": "Handoff Study protocol v1.2 (synthetic)\n",
    "protocol/deviation_log.csv": "id,date,description,outcome_seen\nD01,2026-03-02,P10 prior exposure,yes\n",
    "data/synthetic/decisions_schema.json": '{"participant": "str", "condition": "str", "errors": "int"}\n',
    "code/ch05_analysis.py": "print('analysis placeholder')\n",
    "PERMISSIONS.md": "T3 decision data: released under data-use agreement only.\n",
}

def sha256(path):
    return hashlib.sha256(path.read_bytes()).hexdigest()

def build(root):
    for rel, text in FILES.items():
        p = root / rel
        p.parent.mkdir(parents=True, exist_ok=True)
        p.write_bytes(text.encode("utf-8"))  # bytes: identical hashes on every OS
    lines = [f"{sha256(root / rel)}  {rel}" for rel in sorted(FILES)]
    (root / "MANIFEST.sha256").write_bytes(("\n".join(lines) + "\n").encode("utf-8"))
    return lines

def verify(root):
    bad = []
    for line in (root / "MANIFEST.sha256").read_text(encoding="utf-8").splitlines():
        digest, rel = line.split("  ", 1)
        if sha256(root / rel) != digest:
            bad.append(rel)
    return bad

if __name__ == "__main__":
    with tempfile.TemporaryDirectory() as tmp:
        root = Path(tmp)
        for line in build(root):
            print(line[:16] + "...  " + line.split("  ", 1)[1])
        print("verify (untouched):", verify(root) or "all match")
        (root / "protocol/deviation_log.csv").write_bytes(b"id,date,description,outcome_seen\n")
        print("verify (after edit):", verify(root))

Expected output (verified with Python 3; files are written as bytes so the digests are identical on every operating system):

5b99143807791992...  PERMISSIONS.md
56a6b195fb7d31dc...  code/ch05_analysis.py
97d916df3e33a455...  data/synthetic/decisions_schema.json
73a36f3f76dd754b...  protocol/deviation_log.csv
419c6fe74043bbaa...  protocol/protocol_v1.2.md
verify (untouched): all match
verify (after edit): ['protocol/deviation_log.csv']

Interpretation. A manifest makes silent changes visible. The edit to the deviation log — the very file that records departures from protocol — is detected immediately. In a real archive, any legitimate change produces a new release with a regenerated manifest and a change-log entry explaining it.

The candidate's README states plainly: "This archive lets you run every analysis on schema-matched synthetic data and regenerate every reported aggregate from released aggregate files. Participant-level decision data are available only under a data-use agreement, as permitted by participants' consent (see PERMISSIONS.md). No incident records from any organization were used. Scenario materials are synthetic and derived from the OpenCyber Academy Harbor Library, Cedar Training Network and Lumen Library Reservations teaching cases."

8.8 Practical activity: archive audit

Ask a colleague who has not worked on the project to audit the archive using only its README. They should record: whether they could run the pipeline; whether each reported number could be regenerated or was explained as restricted; whether licenses and permissions were clear; and every question they had to ask. Each question becomes a documentation fix before release.

For the SHA archive, the decisive test is whether the primary estimate can be regenerated from hashed raw rater sheets by script, and whether restricted items such as consented session recordings are explained rather than silently missing.

8.9 A specimen archive layout with documentation

Section 8.3.1 described the structure of the reproducibility archive. This section gives a fuller specimen for the Handoff Study, combining the reproducibility rules of Sandve et al. (2013), the documentation formats of Gebru et al. (2021) and Mitchell et al. (2019), and the FAIR principles of Wilkinson et al. (2016).

handoff-study-archive/
  README.md                 entry point: what, how to run, what is missing and why
  LICENSE-code              e.g., an OSI-approved license for code
  LICENSE-docs              e.g., a Creative Commons license for text and synthetic data
  CITATION.cff              how to cite the archive
  MANIFEST.sha256           hash of every file (Chapter 8 lab)
  protocol/
    protocol_v1.0.pdf       as approved by the ethics board
    protocol_changes.md     every amendment with date, reason, and before/after
    preregistration.pdf     time-stamped registration
  materials/
    scenarios/              scenario scripts (synthetic; safe to share)
    sha_template.md         the artifact itself
    rater_rubric.md         scoring rubric with worked examples
    expert_panel.md         defensible-option elicitation procedure and dropped points
  data/
    DATASHEET.md            datasheet: collection, composition, labeling, known shifts, uses
    synthetic/              fixture with the same schema as the restricted data
    restricted/ACCESS.md    how qualified researchers can request access; no data here
  analysis/
    environment.lock        exact versions of interpreter and packages
    run_all.sh              one command: fixture → every number in the thesis
    primary.py, secondary.py, multiverse.py, metaanalysis.py
    outputs/                generated tables and figures with the commit that produced them
  critique/
    internal_review.md      objection list and responses (Chapter 7)
    reviewer_letters/       journal exchanges, with permission
  governance/
    data_management_plan.md roles, retention, destruction, access decisions
    incident_log.md         any data incidents involving study data

Three choices in this specimen deserve justification.

A separate restricted-access path. The restricted decision data cannot be published. The archive therefore states exactly what exists, who holds it, and under what conditions it can be accessed, and it provides a synthetic fixture with the same schema so that run_all.sh executes end to end. This places the archive at a defined point on Peng's spectrum (Peng, 2011): code and synthetic data are fully executable; the real data are findable and accessible under conditions, satisfying the "F" and conditional "A" of FAIR without violating consent.

Changes as first-class files. protocol_changes.md turns deviations into documentation. A reader comparing the pre-registration with the final analysis should not need to infer differences.

Critique in the archive. Including the internal objection list and reviewer exchanges is unusual but valuable: it lets future researchers see which threats were raised and how they were addressed, which is part of the evidence for the claim.

8.10 Long-term stewardship of incident-research data

Incident-research data often outlive the doctoral project and remain sensitive. Section 8.4 listed the elements of a governance plan. In practice, three failure patterns recur:

1. Orphaned data. The candidate graduates; the restricted data remain on a departmental server with no steward. Requests go unanswered, or worse, access controls lapse. Remedy: name a steward by role (not only by person), with a successor, in the data management plan.
2. Consent drift. Later researchers request the data for purposes the original consent did not cover. Remedy: record the consent scope in machine-readable form in ACCESS.md and require requesters to state their purpose against it.
3. Re-identification risk growth. Data de-identified in one year may become re-identifiable as auxiliary data accumulate — the lesson of Narayanan and Shmatikov (2008) and Sweeney (2002) discussed in Chapter 1. Remedy: schedule periodic re-assessment and retain the authority to withdraw or further restrict access.

Retention should be set by the consent terms and institutional policy, not by convenience. When the retention period ends, destruction should be logged in incident_log.md or a governance log, with the manifest updated to show which files no longer exist.

8.11 Writing the thesis for two audiences

A thesis is examined by researchers but read, if it is useful, by practitioners. The research audience needs the argument (§8.1), the evidence and its limits. The practitioner audience needs to know what to do differently, under what conditions, and at what cost. A short practitioner summary — two pages, written after the thesis is complete — should state:

• The claim in operational terms ("for shift handoffs during multi-day investigations with incomplete telemetry…").
• The size of the expected benefit and its range for a new organization (the prediction interval of §6.10, not only the average).
• The cost (sender time per handoff) and the known risks (no evidence of increased harmful actions within the declared margin; untested for cross-organization handoffs).
• How to evaluate adoption locally — for example, by sampling handoffs before and after adoption and scoring them for transmitted hypotheses and gaps (§6.12).

Writing the practitioner summary is also a test of the thesis: if the summary cannot be written without overstating, the thesis's claims may be overstated too.

8.12 The defense as a final critique round

The defense is the last structured critique before the claim enters the literature. Preparing for it reuses the materials of earlier chapters:

• The objection inventory (EX050) draws on the critique table in §7.10, the limitation table in §8.5 and the rival-account table in §3.14.
• The claim-evolution table (EX046) shows examiners that narrowing was evidence-driven.
• The archive lets an examiner verify any number on the spot: "which script produced Table 5.3?" should be answerable by pointing to a file and commit.

Expect examiners to probe the joints of the argument where theory meets measurement: whether rater-scored errors measure situation awareness in Endsley's sense (Endsley, 1995), whether expert decision processes described by naturalistic decision-making research (Klein, 2008) were represented in the sample, and whether the prediction interval supports any adoption recommendation. The candidate who has already written these answers into the thesis will find the defense a discussion rather than an interrogation.

8.13 What remains open in incident-response science

A doctoral thesis ends with future work; a book on doctoral research in incident response can end with the field's open questions, as they appear from the literature reviewed here.

1. Valid, shareable outcome measures. The field lacks widely accepted measures of response decision quality that can be used across organizations without exposing sensitive data.
2. Cross-organizational coordination evidence. Crisis-management research (Comfort, 2007) and command-system research (Bigley and Roberts, 2001) offer frameworks; controlled evidence about cyber-incident coordination across organizations is scarce.
3. Human–AI response teams. The reliance literature predicts misuse and disuse; incident-response studies that measure both, with seeded model errors and long time horizons, are rare.
4. Longitudinal and field evidence. Most controlled evidence is from single-session simulations. Studies that follow an intervention through months of operation, even with weaker designs, would test whether simulation effects persist.
5. Theory that makes risky predictions. The traditions in §2.9 provide candidate mechanisms; few security studies test them against each other. Designs that discriminate among accounts would turn the field's growing empirical base into cumulative knowledge.

Each of these is a thesis-sized problem. A candidate who finishes the Handoff Study is well placed to start the next one.

Common errors and limits

• Organizing the thesis chronologically rather than as an argument.
• Presenting only the final claim and hiding how it was narrowed.
• An archive that contains code but no description of what is missing and why.
• Relicensing restricted data or participant material beyond consent.
• No custodian or access procedure after the candidate leaves.
• Limitations listed without affected claims or resolution paths.
• Limit: this chapter describes archive practice in general terms; specific repositories, identifiers and institutional requirements must be checked at the time of deposit.

Exercises — chapter 8

EX045 — Argument map. Objective: map the thesis argument. Inputs: the Handoff Study's final claim. Constraints: use all six argument elements in §8.1; each element cites a chapter section. Steps: (1) write the final claim; (2) fill grounds, warrant, backing, qualifier and rebuttal in a table; (3) attach a chapter section to each cell; (4) mark any cell whose backing is weaker than the claim needs. Expected evidence: argument table. Rubric: 3 = complete, traceable, qualifiers and rebuttals honest; 2 = one element weak; 1 = claim and grounds only; 0 = none.

EX046 — Claim-evolution table. Objective: document narrowing. Inputs: Chapters 3–7. Constraints: at least four stages; each change linked to specific evidence. Steps: (1) extract the claim as stated in Chapters 3, 5, 6 and 7; (2) tabulate them in order; (3) for each change, cite the result that caused it; (4) write the explanation. Expected evidence: table and 100-word explanation of why the trajectory strengthens the thesis. Rubric: 3 = complete with evidence links and a sound explanation; 2 = missing a stage; 1 = no evidence links; 0 = none.

EX047 — Archive README. Objective: write an honest archive entry point. Inputs: the structure in §8.3.1. Constraints: 300–500 words; include run instructions, what is not included and why, licenses, permissions and how to report problems. Steps: (1) list the archive's top-level folders; (2) write a quick-start command sequence and test it; (3) list every excluded item with the reason; (4) add licenses, permissions and a contact route. Expected evidence: README. Rubric: 3 = a stranger could run it and understand every gap; 2 = one element missing; 1 = code-only README; 0 = none.

EX048 — Manifest lab. Objective: verify integrity. Inputs: the script in §8.7. Constraints: offline. Steps: (1) run and confirm the output; (2) modify the script so that it also detects a file added after the manifest was built (e.g., data/synthetic/extra.csv); (3) run and record the output. Expected evidence: modified code and output. Rubric: 3 = detects added files without false positives on the manifest itself, outputs recorded; 2 = detects but flags the manifest; 1 = not working; 0 = none.

EX049 — Data-governance plan. Objective: plan post-submission stewardship. Inputs: the Handoff Study's restricted decision data. Constraints: cover the five elements of §8.4. Steps: (1) inventory the restricted data and its consent terms; (2) assign a steward and a successor; (3) define access, retention and destruction procedures; (4) define how requests and incidents involving the data are handled. Expected evidence: one-page plan. Rubric: 3 = named roles, concrete procedures and retention consistent with consent; 2 = one element vague; 1 = generic; 0 = none.

EX050 — Defense objection inventory. Objective: prepare for the strongest objections. Inputs: the whole book. Constraints: at least eight objections spanning integrity, measurement, external validity, ethics and reproducibility. Steps: list each objection, a short evidence-based answer and the location of supporting material. Expected evidence: inventory. Rubric: 3 = eight or more serious objections with honest answers, including concessions; 2 = some answers defensive; 1 = trivial objections; 0 = none.

Chapter synthesis

A thesis in incident-response research is an argument whose claims are exactly as strong as their evidence, with the narrowing of those claims documented rather than hidden. Its reproducibility archive makes the evidence inspectable within legal and ethical limits, with provenance, integrity manifests, explicit permissions and a plan for stewardship after submission. The Handoff Study ends with a narrow, useful and honest claim, an archive that a stranger can audit, and a list of future studies that begin where its limits do. That is the standard this book has aimed to teach.

Continue studying — chapter 08 of 08

← Chapter 07 · Course chapter index · Next course via chapter index →

This is the final chapter of this course; the index links the next course in the full study path.