Chapter 7 — Thesis synthesis and reproducibility archive

Lesson map

Lesson 07 published. Map, example, EX034.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["7.1 The thesis as an argument"]
  N2["7.2 Monograph or thesis by publication"]
  N3["7.3 Structure of the Cascade Study thesis"]
  N4["7.4 The reproducibility archive"]
  N5["7.5 Synthetic data: invented versus calibrated"]
  N6["7.6 Documenting models, data and studies"]
  N7["7.7 Writing limitations"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX034"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["7.1 The thesis as an argument"]
  N2["7.2 Monograph or thesis by publication"]
  N3["7.3 Structure of the Cascade Study thesis"]
  N4["7.4 The reproducibility archive"]
  N5["7.5 Synthetic data: invented versus calibrated"]
  N6["7.6 Documenting models, data and studies"]
  N7["7.7 Writing limitations"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX034"]

[Learning outcomes and connection]
    |
    v
[7.1 The thesis as an argument]
    |
    v
[7.2 Monograph or thesis by publication]
    |
    v
[7.3 Structure of the Cascade Study thesis]
    |
    v
[7.4 The reproducibility archive]
    |
    v
[7.5 Synthetic data: invented versus calibrated]
    |
    v
[7.6 Documenting models, data and studies]
    |
    v
[7.7 Writing limitations]
    |
    v
[Worked example] --> [Exercise EX034]

Learning outcomes and connection

By the end of this chapter the learner will be able to:

1. Structure a thesis as an argument in which every claim is linked to evidence, warrant, scope and limitation.
2. Decide between a monograph and a thesis by publication and integrate either into one coherent argument.
3. Design a tiered reproducibility archive that releases as much as is lawful and safe, under infrastructure-information protection.
4. Distinguish invented synthetic data from synthetic data calibrated to real places, and assess residual disclosure risk.
5. Document models, datasets and simulation studies with recognized structures.
6. Write a limitations section that informs rather than apologizes.

Chapter 6 produced the narrowed claim. This chapter assembles the thesis around it and builds the archive that lets others check it without exposing the infrastructure it studies.

7.1 The thesis as an argument

A thesis is not a report of everything done; it is an argument for a contribution. A useful scaffold is Toulmin's analysis of arguments (Toulmin, 1958/2003): a claim supported by data (evidence) through a warrant (the reason the evidence supports the claim), with backing for the warrant, a qualifier (how strongly) and rebuttals (conditions under which the claim fails). Applied to infrastructure research:

Lesson table; scroll horizontally to read all columns →

Each chapter of the thesis should contribute an identifiable element. A chapter that supplies none is a candidate for an appendix.

7.2 Monograph or thesis by publication

A monograph presents one integrated argument; a thesis by publication collects papers with an integrating introduction and conclusion. Infrastructure theses often mix genres — a model paper, an empirical study of operators, a policy analysis — which suits publication-based theses, but the integrating chapters must do real work: state the overarching claim, show how each paper contributes data or backing, reconcile differences in scope between papers and present limitations across the whole. Regulations about format, co-authorship statements and prior publication differ between institutions; this educational track does not grant degrees and cannot advise on any institution's rules beyond pointing out that they must be checked early.

7.3 Structure of the Cascade Study thesis

1. Introduction. Problem, gap, contribution types, overarching claim and its scope.
2. Background. Definitions, interdependency and resilience literature, restoration practice, scheduling theory (Chapters 1–2 of this book).
3. Method. Mechanism, hypotheses, baselines, protocol, ethics and information protection (Chapter 3).
4. Verification and baselines. Reproductions, verification tests, elicitation and pilot (Chapter 4).
5. Main study and validation. Simulations, tabletop, validation, second region (Chapter 5).
6. Critique and boundaries. Red-team tests, sensitivity, many-region analysis, narrowed claim, dual-use review (Chapter 6).
7. Discussion. What practitioners and policy-makers can do with the narrowed claim; what remains unknown.
8. Limitations.
9. Conclusion.
10. Appendices. Preregistration and amendments, deviation log, verification test listing, elicitation protocols, tabletop materials, archive manifest.

7.4 The reproducibility archive

7.4.1 Goals and constraints

Reproducibility in this field must satisfy two goals that pull in opposite directions: enabling others to check the work, and not publishing information that helps someone harm infrastructure or identifies people. The FAIR principles — findable, accessible, interoperable, reusable — help here because FAIR does not mean open: data can be findable and its access conditions clearly described while access itself is controlled.

7.4.2 Release tiers

Lesson table; scroll horizontally to read all columns →

The archive's manifest lists every file with a cryptographic hash and its tier, so that anyone can verify that the public files are the ones used, and examiners can see what exists at higher tiers.

7.4.3 Practical activity: building a tiered manifest

# Lab 7 - build a tiered reproducibility manifest for a synthetic thesis archive (offline)
import hashlib, json, os, tempfile
files = {
    "code/lab5_mc.py": b"# simulation code\n",
    "data/synthetic_riverbend_params.csv": b"feeder,customers\nF1,9000\nF2,1200\nF3,2500\nF4,6000\n",
    "data/partner_feeder_map.csv": b"feeder,substation,gps\nF2,SUB-7,redacted\n",
    "results/mc_summary.json": b'{"mean_dloss": -46.86}\n',
    "docs/operator_interview_notes.txt": b"P03: we restore by customer count\n",
}
RULES = [  # first matching rule wins; order matters
    ("partner_", "T3-withheld: partner-owned infrastructure detail; describe, do not release"),
    ("interview", "T2-controlled: human-subjects data; release only under approved access agreement"),
    ("synthetic_", "T0-public"),
    ("code/", "T0-public"),
    ("results/", "T0-public"),
]
def tier(path):
    for key, label in RULES:
        if key in path:
            return label
    return "UNCLASSIFIED: stop and review"
with tempfile.TemporaryDirectory() as root:
    manifest = []
    for rel, content in sorted(files.items()):
        p = os.path.join(root, rel)
        os.makedirs(os.path.dirname(p), exist_ok=True)
        with open(p, "wb") as fh:
            fh.write(content)
        manifest.append({"path": rel, "sha256": hashlib.sha256(content).hexdigest()[:16], "tier": tier(rel)})
    print(json.dumps(manifest, indent=1))
    print("public files:", sum(m["tier"].startswith("T0") for m in manifest), "of", len(manifest))

Expected output:

[
 {
  "path": "code/lab5_mc.py",
  "sha256": "e9db014f8aeb90da",
  "tier": "T0-public"
 },
 {
  "path": "data/partner_feeder_map.csv",
  "sha256": "160cefe894e75af9",
  "tier": "T3-withheld: partner-owned infrastructure detail; describe, do not release"
 },
 {
  "path": "data/synthetic_riverbend_params.csv",
  "sha256": "66bd4b5c8f37d273",
  "tier": "T0-public"
 },
 {
  "path": "docs/operator_interview_notes.txt",
  "sha256": "35c6e20265ce92c5",
  "tier": "T2-controlled: human-subjects data; release only under approved access agreement"
 },
 {
  "path": "results/mc_summary.json",
  "sha256": "7bde71d2f02e64ad",
  "tier": "T0-public"
 }
]
public files: 3 of 5

Two design choices deserve attention. First, the rules are ordered and the first match wins: if the "synthetic" rule were placed before the "partner" rule, a file named synthetic_partner_feeders.csv would be published. Order the most restrictive rules first. Second, an unmatched file returns "UNCLASSIFIED: stop and review" rather than a default tier; a release pipeline should fail closed. Real pipelines would also scan file contents (coordinates, asset identifiers, names), not only paths.

7.5 Synthetic data: invented versus calibrated

The regions in this book are invented: their numbers were chosen to teach, and they correspond to no place. A thesis based on a real partner would more likely produce calibrated synthetic data — generated to match statistical properties of real data. Calibration preserves research value and also preserves some disclosure risk: if the real region has a single hospital on a small feeder with a short-buffer water plant, a calibrated synthetic region that preserves those relations may still point to it. Assess residual risk explicitly: Could an informed person match the synthetic configuration to a real one? Would that match reveal anything not already public? Generalizing (ranges instead of values), perturbing, and mixing several real configurations reduce risk; the assessment and the chosen measures belong in the archive documentation.

7.6 Documenting models, data and studies

• Models. For agent-based components, the ODD protocol — overview, design concepts, details — gives a standard structure (Grimm et al., 2006). For any simulator, document purpose, entities and state variables, processes and scheduling, inputs and their sources, verification tests and known limitations.
• Datasets. Datasheets record motivation, composition, collection process, preprocessing, uses, distribution and maintenance (Gebru et al., 2021).
• Learned components. If the study includes a learned model (for example, a detector), a model card records intended use, evaluation conditions and limitations (Mitchell et al., 2019).
• Computation. Pin language and library versions, record seeds, provide a single command that regenerates every table and figure, and include the verification tests. Sandve et al. (2013) give ten practical rules.

7.7 Writing limitations

A limitations section is part of the argument: it states the rebuttal conditions and their likely direction. Weak limitations sections list generic caveats ("more research is needed"). Strong ones say, for each limitation, what it is, how it could change the result and in which direction, and what study would resolve it. For the Cascade Study:

Lesson table; scroll horizontally to read all columns →

7.8 Worked example: assembling the archive

The candidate's first archive plan placed everything at T2 "to be safe". The committee rejected it: over-restriction prevents verification and is not safe, only unaccountable. The revised plan used the tier table in §7.4.2, published all code and synthetic configurations at T0 with a single regeneration command, placed tabletop transcripts at T2 under the consent terms participants signed, and recorded the T3 partner material in a description that told examiners exactly which analyses depended on it — none of the confirmatory results did, by design, because the study used invented regions for all public claims and used partner data only for elicitation ranges and face validity. That design decision, made in Chapter 1, is what made a largely public archive possible.

7.9 Common errors and limits

• Writing the thesis as a chronological lab notebook rather than an argument.
• Restricting everything, or publishing everything, instead of tiering.
• Equating FAIR with open.
• Treating calibrated synthetic data as automatically safe.
• Release scripts that fail open.

• Limitations sections without direction or resolving studies.

• Placing everything at a restrictive tier to be safe, which prevents verification and makes the work unaccountable rather than safe.

• Ordering release rules so that a permissive pattern such as synthetic matches before a restrictive one such as partner.
• Classifying files by name alone, without scanning contents for coordinates, asset identifiers or names.
• Letting confirmatory results depend on withheld partner material, so examiners cannot check them.
• Omitting a single command that regenerates every table and figure from the public files.

Exercises — chapter 7

EX039 — Argument map. Objective: expose the thesis argument. Inputs: the narrowed claim of §6.5. Constraints: use all six Toulmin elements. Steps: (1) map the claim; (2) mark which thesis chapter supplies each element; (3) identify the weakest warrant and the evidence that would strengthen it. Expected evidence: map and paragraph. Rubric: 3 = complete map, weakest warrant correctly identified (model adequacy for real regions, or omitted human buffers) with a specific strengthening study; 2 = one element thin; 1 = claim and data only; 0 = none.

EX040 — Tier assignment. Objective: tier a realistic archive. Inputs: ten artifacts: simulator code; verification tests; Riverbend configuration; elicitation forms with partner names; aggregated elicitation ranges; tabletop audio; coded rationales; consent forms; a partner's restoration procedure; final figures. Constraints: use the four tiers; justify each. Steps: (1) assign each artifact a tier; (2) justify the tier in one line; (3) propose a transformation that would move each restricted item to a more open tier. Expected evidence: table. Rubric: 3 = sensible tiers (code, tests, configurations and figures T0; aggregated ranges T1; coded rationales T2 or T1 if de-identified; audio, consent forms and named elicitation forms T2 or T3; partner procedure T3) with transformations; 2 = one or two misjudged; 1 = uniform tiering; 0 = none.

EX041 — Fail-closed manifest. Objective: improve Lab 7. Inputs: Lab 7. Constraints: offline. Steps: (1) add a file data/synthetic_partner_feeders.csv; (2) show that the current rule order classifies it correctly and explain why; (3) move the synthetic_ rule first, re-run and record the failure; (4) add a content check that refuses public tier for any file containing the string SUB-. Expected evidence: code, outputs, explanation. Rubric: 3 = demonstrates the ordering hazard, the content check works, explanation links to fail-closed design; 2 = partial; 1 = not run; 0 = none.

EX042 — Residual disclosure assessment. Objective: assess calibrated synthetic data. Inputs: a synthetic region calibrated to a real county with one hospital, two water plants and five feeders, preserving which feeder hosts the hospital. Constraints: use the questions in §7.5. Steps: (1) assess the re-identification risk; (2) assess what a match would reveal; (3) propose transformations and evaluate their cost to research value. Expected evidence: one-page assessment. Rubric: 3 = recognizes re-identification through unique structure, proposes mixing or generalization, weighs research value honestly; 2 = assessment shallow; 1 = declares synthetic safe; 0 = none.

EX043 — Model documentation. Objective: document the Lab 5 simulator. Inputs: Lab 5 code. Constraints: follow the model elements in §7.6; at most two pages. Steps: (1) write purpose, entities, processes and scheduling; (2) list inputs and their sources; (3) list the verification tests; (4) list known limitations, including at least three from §7.7. Expected evidence: document. Rubric: 3 = accurate to the code (event-based intervals, restart lag, crew assignment, OR semantics for dependencies), limitations specific; 2 = minor inaccuracies; 1 = generic; 0 = none.

EX044 — Limitations section. Objective: write limitations that inform. Inputs: your own study or the Cascade Study. Constraints: at least five limitations, each with effect, direction and resolving study. Steps: (1) list the limitations; (2) state the effect and direction of each; (3) name the study that would resolve each; (4) rank them by how much they could change the main claim. Expected evidence: table and ranking paragraph. Rubric: 3 = specific, directional, with resolving studies and a justified ranking; 2 = directions missing for some; 1 = generic caveats; 0 = none.

Chapter synthesis

The thesis is an argument: claim, data, warrant, backing, qualifier and rebuttals, each supplied by an identifiable part of the work. Infrastructure research must reconcile reproducibility with protection of the infrastructure studied and of the people involved; tiered archives with hashed manifests do this, and FAIR principles describe access without implying openness. Release tooling should fail closed and check content, not only names. Invented synthetic data carry little disclosure risk; calibrated synthetic data need an explicit residual-risk assessment. Document models, data and computation with recognized structures. Limitations state direction and resolving studies, which turns them into the next research agenda.