Chapter 7 — Thesis synthesis and reproducibility archive
Lesson map
Lesson 07 published. Map, example, EX039.
flowchart TD
N0["Learning outcomes"]
N1["7.1 The argument"]
N2["7.2 Confirmatory versus exploratory"]
N3["7.3 The archive"]
N4["7.4 Testing the archive"]
N5["7.5 Licensing and privacy"]
N6["7.6 Implications"]
N7["7.7 Mapping to frameworks"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX039"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["7.1 The argument"]
N2["7.2 Confirmatory versus exploratory"]
N3["7.3 The archive"]
N4["7.4 Testing the archive"]
N5["7.5 Licensing and privacy"]
N6["7.6 Implications"]
N7["7.7 Mapping to frameworks"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX039"]
[Learning outcomes]
|
v
[7.1 The argument]
|
v
[7.2 Confirmatory versus exploratory]
|
v
[7.3 The archive]
|
v
[7.4 Testing the archive]
|
v
[7.5 Licensing and privacy]
|
v
[7.6 Implications]
|
v
[7.7 Mapping to frameworks]
|
v
[Worked example] --> [Exercise EX039]
Learning outcomes
You will (1) synthesise findings into a coherent thesis argument; (2) build a reproducibility archive; (3) separate confirmatory and exploratory results; (4) state implications for practice and research.
7.1 The argument
The thesis argues: (1) benign triggers dominate analyst workload, consistent with Alahmadi et al. (2022); (2) context on alert cards improves triage accuracy; (3) benign-history context carries a specific risk of dismissing true attacks; (4) alert design should pair context with counter-evidence prompts. Each step rests on a different kind of evidence: the first on literature, the second on the registered experiment, the third on the failure analysis and exploratory follow-up, and the fourth on Nadia's judgement from all three. The thesis labels the fourth as a recommendation, not a finding.
7.2 Confirmatory versus exploratory
Chapter results are tagged: the accuracy effect is confirmatory (registered); the field attribution and time effect are exploratory. Tables mark each. Tagging tells readers how much weight each result can bear. The confirmatory accuracy effect was tested as registered; the attribution to the benign-history field comes from a small follow-up with four analysts and is a lead, not a conclusion. Nadia's summary table repeats each tag next to the archive file that reproduces the result. The time result carries a secondary tag, matching how it was registered before the main study.
7.3 The archive
The archive contains: the alert generator and seed; the 320-alert corpus with labels; Sigma-format rules used; anonymised analyst logs; analysis scripts with pinned versions; the pre-registration and change log; and a README explaining how to rerun every table. The archive lets another researcher regenerate every alert, rule match, card and table from the seed. Anonymised logs replace the raw triage records, as the consent forms promised. The change log and pre-registration show readers which choices were fixed before the data and which came later.
7.4 Testing the archive
A fellow student downloads the archive on a clean machine and regenerates all tables, using only the archive's instructions. Two figures differ in the fourth decimal because of a rounding call; Nadia fixes the script and notes it. The differences were small, but finding them shows that the test checks the archive itself rather than Nadia's own machine. Nadia records the fix in the change log and tags a new archive version, and the student's second run on the clean machine makes the outputs match exactly.
7.5 Licensing and privacy
Corpus and code are released under an open licence. Analyst logs are anonymised: analyst IDs are random, timestamps shifted, free-text notes removed after review for identifying content. Removing free-text notes after review prevents analysts from being identified by their writing or by details of their workplace. Shifted timestamps keep durations accurate while hiding when sessions occurred. The open licence lets other researchers reuse the corpus for different alert-card experiments, which is the point of the first contribution. The archive README explains each anonymisation step, so reusers know which fields were removed and why.
7.6 Implications
For SOCs: test enrichment fields one at a time and monitor missed-attack rates; history fields need guard rails. For research: field-level experiments and realism studies on corpora. The SOC-CMM (van Os, 2016) offers a way for SOCs to place such changes within capability maturity. The implications follow the evidence: the missed attacks support the advice on history fields, while the exploratory attribution supports testing fields one at a time. Each implication in the thesis names the result it rests on, so a reader can follow it back to the evidence and its confirmatory or exploratory tag before applying it at their own SOC.
7.7 Mapping to frameworks
Findings are mapped to NIST CSF 2.0's Detect function (adverse event analysis) so practitioners can locate them. Mapping the work to adverse event analysis lets a SOC manager see that the findings concern how alerts are assessed after detection, not how rules are written. Nadia also notes the related Respond activities, since dismissing a true attack at triage delays every later step of incident handling. The mapping appears in the practitioner article as well as the thesis. The thesis cites NIST CSF 2.0 for the mapping, using its Detect and Respond functions by name.
Worked example
Nadia's thesis summary table lists each claim, evidence chapter, status (confirmatory or exploratory) and archive file that reproduces it.
Synthetic case. The archive and test are invented.
Errors and limits
Archives decay as software changes; pinned versions and a container description slow but do not stop decay.
Exercises
- EX039 — Four claims. State the thesis's four claims. Rubric: 3 = benign triggers dominate; context helps; history risk; pair with counter-prompts; 2 = three; 1 = one; 0 = none.
- EX040 — Status tags. Which results are confirmatory and which exploratory? Rubric: 3 = accuracy confirmatory; attribution and time exploratory; 2 = partial; 1 = all confirmatory; 0 = none.
- EX041 — Archive contents. List six archive items. Rubric: 3 = six of generator, corpus, rules, logs, scripts, registration, README; 2 = four; 1 = two; 0 = none.
- EX042 — Archive test. What did the clean-machine test reveal? Rubric: 3 = fourth-decimal rounding difference; fixed and noted; 2 = partial; 1 = vague; 0 = none.
- EX043 — Log anonymisation. How were analyst logs anonymised? Rubric: 3 = random IDs, shifted times, free text removed; 2 = two; 1 = one; 0 = none.
- EX044 — Practice implication. Give one implication for SOCs. Rubric: 3 = test fields singly, monitor misses, guard history fields; 2 = partial; 1 = vague; 0 = none.
Synthesis
A thesis earns trust by a clear argument, honest status tags and an archive that others can actually rerun.
Continue studying — chapter 07 of 08
← Chapter 06 (draft) · Course chapter index · Chapter 08 (draft) →