Chapter 8 — Independent research review and defense
Lesson map
Lesson 08 published. Map, example, EX045.
flowchart TD
N0["Learning outcomes and connection"]
N1["8.1 How examiners read"]
N2["8.2 Families of questions"]
N3["8.3 Reproduction by an independent reader"]
N4["8.4 What if you had chosen differently? — the specif"]
N5["8.5 Counterexamples and how to present them"]
N6["8.6 Preparing for the defense"]
N7["8.7 After the defense: corrections"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX045"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["8.1 How examiners read"]
N2["8.2 Families of questions"]
N3["8.3 Reproduction by an independent reader"]
N4["8.4 What if you had chosen differently? — the specif"]
N5["8.5 Counterexamples and how to present them"]
N6["8.6 Preparing for the defense"]
N7["8.7 After the defense: corrections"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX045"]
[Learning outcomes and connection]
|
v
[8.1 How examiners read]
|
v
[8.2 Families of questions]
|
v
[8.3 Reproduction by an independent reader]
|
v
[8.4 What if you had chosen differently? — the specif]
|
v
[8.5 Counterexamples and how to present them]
|
v
[8.6 Preparing for the defense]
|
v
[8.7 After the defense: corrections]
|
v
[Worked example] --> [Exercise EX045]
Learning outcomes and connection
After this chapter you should be able to:
- Describe how examiners read a dissertation and the families of questions they ask.
- Prepare a reproduction package and support an independent reader in reproducing your key results.
- Answer "what if you had chosen differently?" with a specification curve rather than an assertion.
- Find and present counterexamples to your own claims before an examiner does.
- Conduct a defense, record corrections with an audit trail, and share results responsibly with participants, partners and the profession.
The chapter builds on the dissertation of Chapter 7 and closes the course.
8.1 How examiners read
Examiners rarely read a dissertation from front to back on the first pass. A common pattern is: abstract and introduction (what is claimed?), conclusion (does it match?), methods (could this design support it?), results tables (do the numbers support it?), limitations (does the author know the weaknesses?), and only then the literature review and discussion in full. They mark places where a claim is larger than its evidence, where a number cannot be traced, and where an obvious rival was not addressed. Write for that reader: put the answer to each research question where it can be found, make every number traceable, and address the rivals explicitly.
Master's examination in many systems includes an internal and an external examiner and sometimes an oral examination (viva or defense). The criteria usually resemble those in the front matter of this book: a clear question, a fitting design, competent and reproducible analysis, conclusions within the evidence, and a critical understanding of limits. Few examiners expect a master's study to be flawless; they expect you to know where its flaws are and what they do.
8.2 Families of questions
| Family | Typical question | What a good answer does |
|---|---|---|
| Motivation | "Why does under-triage matter more than speed?" | Links to the decision-maker's problem and the asymmetry of consequences (Chapter 2) |
| Design | "Why a crossover? What about carryover?" | States the counterfactual, the assumptions, the checks and their results |
| Measurement | "How do you know the reference standard is right?" | Describes panel independence, the split cases, and the alternative-reference analysis |
| Analysis | "Why per-analyst differences rather than a logistic mixed model?" | Explains clustering, transparency and the estimand; concedes a mixed model is a valid alternative and reports whether it would change conclusions |
| Robustness | "What if you had defined under-triage differently?" | Shows the specification curve |
| Rivals | "Isn't this just the rubric making everyone rate higher?" | Cites the over-triage result and the noisy-case null |
| Transfer | "Would this work in a real SOC at 3 a.m.?" | Gives the directional limitation and what would be needed to test it |
| Contribution | "What is new here?" | States the specific gap from the concept matrix and what the study adds, without inflating it |
| Reflexivity | "You designed the rubric. How did that affect the study?" | Acknowledges allegiance, names the safeguards (independent panel, predeclared plan) and their limits |
8.3 Reproduction by an independent reader
Before submission, ask someone outside the project — another student, a research software engineer — to reproduce the key results from your package without your help. Give them the README, code, the synthetic or restricted dataset and the pre-analysis plan; ask them to regenerate the primary result, the subgroup result and two tables. Record what they could not do. Common failures: undocumented software versions, a path hard-coded to your machine, a seed set in one script and not another, a manual step ("then I removed the duplicate row") that is not in the cleaning log. Each failure is cheaper to fix now than to explain in a defense.
The reader should also try to break your claims: run the analysis with choices you did not make, look for analysts or cases that drive the result, and check whether the result depends on the reference standard. That is the purpose of Lab 8.
8.4 "What if you had chosen differently?" — the specification curve
8.4.1 The idea
Every analysis embodies choices: how to define the outcome, which cases to include, which reference standard to use, how to handle a withdrawal. Different reasonable choices could produce different answers. Multiverse analysis (Steegen et al. 2016) and specification curve analysis (Simonsohn, Simmons and Nelson 2020) run the analysis under every combination of defensible choices and display the distribution of results. The aim is not to find the best specification; it is to show how much the conclusion depends on choices.
Two cautions. The set of specifications must be defensible choices, decided as far as possible before looking — not every conceivable variation. And specifications are not independent evidence: twelve agreeing specifications on the same data are one dataset viewed twelve ways.
8.4.2 Practical activity: the Triage Rubric specification curve
Lab 8 crosses three choices: the reference standard (the panel's consensus, or the minority view for the two cases where the panel split 2–1, C07 and C19, which the minority rated 2 rather than 3), the error threshold (one or two levels below reference) and the case inclusion rule (high-impact only, severity 2 and above, critical only). It also reports a leave-one-analyst-out range, the primary difference by sequence, and a counter-check of the rival "the rubric just inflates every rating". Save it as lab8_review.py next to the Chapter 6 files.
# Lab 8 - an independent reader re-runs the analysis and asks "what if you had chosen differently?" (offline)
# Imports lab6_data.py and lab6_analysis.py. All data synthetic.
import statistics as st
from lab6_analysis import ROWS, paired, boot_ci
DISPUTED = {"C07", "C19"} # panel split 2-1 on reference severity 3; the minority rated them 2
def with_reference(rows, alt):
if not alt:
return rows
return [dict(r, ref=2) if r["case"] in DISPUTED else r for r in rows]
specs = []
for alt in (False, True):
for levels in (1, 2):
for incl_name, keep in (("ref>=3", lambda r: r["ref"] >= 3), ("ref>=2", lambda r: r["ref"] >= 2), ("ref=4", lambda r: r["ref"] == 4)):
d, *_ = paired(with_reference(ROWS, alt), keep=keep, levels=levels)
lo, hi = boot_ci(d, reps=2000)
specs.append((st.mean(d), lo, hi, f"ref={'minority' if alt else 'panel'} under>={levels} {incl_name}"))
specs.sort()
print("specification curve (rubric - usual, under-triage proportion), sorted:")
for m, lo, hi, name in specs:
print(f" {m:+.3f} [{lo:+.3f}, {hi:+.3f}] {'excludes 0' if hi < 0 or lo > 0 else 'includes 0'} {name}")
print(f"specifications: {len(specs)} all negative: {all(m < 0 for m, *_ in specs)} CI excludes 0: {sum(hi < 0 for _, _, hi, _ in specs)}")
d, R, U, both = paired(ROWS)
loo = [st.mean([x for a, x in zip(both, d) if a != drop]) for drop in both]
print(f"leave-one-analyst-out range: {min(loo):+.3f} to {max(loo):+.3f} (primary {st.mean(d):+.3f})")
seq = {}
for a, x in zip(both, d):
seq.setdefault((int(a[1:]) - 1) % 4 + 1, []).append(x)
print("primary difference by sequence: " + " ".join(f"S{k}:{st.mean(v):+.3f} (n={len(v)})" for k, v in sorted(seq.items())))
low = lambda c: [r["rating"] > r["ref"] for r in ROWS if r["cond"] == c and r["ref"] <= 2 and r["analyst"] in both]
print(f"counter-check 'rubric just inflates every rating': over-triage of low-impact cases U {st.mean(low('U')):.3f} R {st.mean(low('R')):.3f}")
Expected output:
specification curve (rubric - usual, under-triage proportion), sorted:
-0.246 [-0.391, -0.101] excludes 0 ref=minority under>=1 ref=4
-0.246 [-0.391, -0.101] excludes 0 ref=panel under>=1 ref=4
-0.174 [-0.245, -0.098] excludes 0 ref=panel under>=1 ref>=3
-0.170 [-0.239, -0.104] excludes 0 ref=panel under>=1 ref>=2
-0.135 [-0.204, -0.070] excludes 0 ref=minority under>=1 ref>=2
-0.118 [-0.193, -0.037] excludes 0 ref=minority under>=1 ref>=3
-0.116 [-0.203, -0.029] excludes 0 ref=minority under>=2 ref=4
-0.116 [-0.203, -0.029] excludes 0 ref=panel under>=2 ref=4
-0.082 [-0.120, -0.049] excludes 0 ref=panel under>=2 ref>=3
-0.065 [-0.096, -0.039] excludes 0 ref=panel under>=2 ref>=2
-0.062 [-0.106, -0.025] excludes 0 ref=minority under>=2 ref>=3
-0.043 [-0.074, -0.017] excludes 0 ref=minority under>=2 ref>=2
specifications: 12 all negative: True CI excludes 0: 12
leave-one-analyst-out range: -0.193 to -0.159 (primary -0.174)
primary difference by sequence: S1:-0.275 (n=5) S2:-0.042 (n=6) S3:-0.229 (n=6) S4:-0.167 (n=6)
counter-check 'rubric just inflates every rating': over-triage of low-impact cases U 0.261 R 0.239
How to read it.
- The curve. All twelve specifications are negative and all intervals exclude zero. The estimates range from −4.3 to −24.6 points. The size depends a great deal on the choices — two-level errors are rarer, so their differences are smaller — but the direction does not.
- Duplicates. The two critical-only specifications are identical under both references, because the disputed cases have reference 3 and are not in the critical-only set. Twelve specifications are effectively ten. Say so; an examiner who notices first will wonder what else was not checked.
- The reference standard matters. Under the minority reference, the high-impact estimate falls from −17.4 to −11.8 points. Both quiet-impact cases where the panel split are cases where the rubric's questions point towards higher severity; if the minority were right, part of the apparent rubric benefit is the rubric agreeing with the majority's way of thinking. This is exactly the construct-validity worry in the limitations table (Chapter 7), now quantified.
- Leave-one-out. Removing any single analyst moves the primary estimate only between −19.3 and −15.9 points: no one analyst drives the result.
- Sequences. The difference varies by sequence, from −27.5 points (S1, five analysts) to −4.2 (S2). With five or six analysts per sequence these are noisy; but S2 — usual on set B first, then rubric on set A — is the smallest, and an examiner may ask whether set A's quiet cases are easier. The dissertation should report this, check the case-set balance (both sets have the same reference distribution and quiet count by design) and treat any case-set explanation as a hypothesis.
- Counter-check. Over-triage of low-impact cases is 0.261 with usual judgement and 0.239 with the rubric: the rubric did not raise every rating.
8.5 Counterexamples and how to present them
A counterexample is a specific case, analyst or subgroup where your claim does not hold. Look for them deliberately: the three analysts who did worse with the rubric; the noisy-case subgroup; sequence S2; the disputed cases. Present each with what it shows and what it does not. "Three of 23 analysts did worse with the rubric; two of them described completing it after deciding (theme 3). This suggests the benefit depends on using the rubric before forming an impression, which the study did not control and which live use may not ensure." A dissertation that presents its own counterexamples is more persuasive than one that hides them, because it shows the author looked.
8.6 Preparing for the defense
8.6.1 The opening summary
Many defenses begin with a short presentation. Use it to answer four questions in about ten minutes: what was the question, what was done, what was found, and what are the limits. Show the dot plot of per-analyst differences, the subgroup result, the cost and the specification curve. Do not re-teach the literature review.
8.6.2 Mock defenses
Hold at least one mock defense with people who did not supervise the project, using the question families in §8.2. Record the questions you answered poorly and prepare short, honest answers. Practise saying "I do not know, but here is how I would find out" and "that is a fair criticism; here is what it does to the conclusion".
8.6.3 Answering
Answer the question asked. Concede valid points quickly and say what they change. Do not defend a choice by its popularity ("everyone uses MTTR"); defend it by its fit to the question. If an examiner proposes an alternative analysis, say whether you ran it (and what happened) or how you expect it would change the result and why.
8.7 After the defense: corrections
Examiners usually require corrections. Keep a corrections table: each requested change, where it was made, and how. Distinguish corrections that change claims from those that change presentation, and re-run the provenance check after any change to numbers. If an examiner's request would require new data collection beyond the scope agreed, discuss it with the supervisor and the examiner rather than inventing data or overstating what can be done.
| Examiner request | Change made | Location | Claim changed? |
|---|---|---|---|
| Report the minority-reference analysis in the main results, not an appendix | Moved specification curve to Results 5.6; added sentence to abstract | §5.6; abstract | Yes: primary claim now qualified by reference standard |
| Explain sequence S2 | Added case-set balance table and a paragraph | §5.7 | No |
| Remove "best practice" from conclusion | Rephrased as conditional implication | §7.2 | Yes: weaker |
8.8 Sharing results responsibly
8.8.1 Participants first
Send participants a plain-language summary of the results before public release, as promised in the information sheet. Do not send individual results to employers, and do not publish anything that combines team, experience and sequence.
8.8.2 Partners and the profession
A briefing for MRSS managers should give the result, its scope and its limits in one page: "In a controlled tabletop, the rubric reduced under-triage of high-impact fictional cases by about 17 points, mostly on quiet-impact cases, at about 45 seconds per case. The effect depends on the reference standard and on using the rubric before deciding. A live pilot with time-stamped rubric completion would be needed before a mandate." For the profession, consider a short paper or conference talk, a preprint, and depositing the case materials and code so others can replicate with their own analysts. A replication with different analysts and a different reference panel is the most valuable next study.
8.8.3 Incidental findings
Research with security practitioners sometimes surfaces information about real weaknesses — a participant mentions an unpatched system, a case writer recognizes their employer's configuration. Such findings are not research data. Do not record or publish them; if they suggest a real risk, the participant or their organization should handle it through its own channels or coordinated disclosure, and the ethics committee should be told if the protocol did not anticipate it.
8.9 Worked example: a mock-defense exchange (synthetic)
Examiner: Your rubric was written by you. The reference panel are senior responders who think like you. Isn't the study measuring how well analysts learn to agree with you?
Student: That is a fair concern, and it is one I tried to measure rather than dismiss. The panel rated cases from their full ground truth before seeing the rubric, and none of them was involved in writing it. But two cases split the panel, and both are quiet-impact cases where the rubric's questions point towards the higher rating. When I use the minority's rating for those two, the primary estimate falls from about 17 to about 12 points, still with an interval excluding zero. So part of the effect may reflect agreement with a way of thinking rather than with an objective truth. The strongest test would be a replication with a reference panel from a different organization, which I recommend.
Examiner: Then why not make the minority analysis primary?
Student: Because the plan fixed the consensus reference before data, and switching after seeing the results would be choosing the analysis by its outcome. I report both prominently, and the abstract now mentions the dependence.
The answer concedes the point, quantifies it, explains the choice by reference to the plan and proposes a test. That is the pattern to practise.
8.10 Next steps after the master's
Some students will continue to doctoral research. The doctoral book in this area (Incident Response doctoral course) asks for an original contribution and independent validation across contexts; a well-run master's study is good preparation, and its open questions (does the effect survive live conditions? does it depend on the reference standard? why do some analysts complete the rubric after deciding?) are the seeds of a doctoral proposal. Others will return to practice with a skill most practitioners lack: the habit of asking what else could explain an improvement, and how one would find out.
8.11 Common errors and limits
- Expecting the examiners to read front to back and burying the answers.
- Submitting a package nobody has tried to reproduce.
- Presenting a specification curve chosen after seeing which specifications favour the claim, or counting duplicate specifications as independent support.
- Hiding counterexamples.
- Defending choices by convention rather than by fit to the question.
- Making corrections without re-running the provenance check.
- Sharing individual results with employers, or publishing incidental security findings.
The limit of review is that reviewers see what you show them. A defense is strongest when you have shown the weak points yourself.
Exercises — chapter 8
EX045 — Run and extend Lab 8. Objective: build and read a specification curve. Inputs: Lab 8. Constraints: offline. Steps: (1) run it; (2) identify the duplicated specifications and say how many distinct ones there are; (3) add a fourth choice — excluding the three analysts with the most negative differences, as a hostile reader might — and report the primary estimate under it (this is a robustness probe, not a legitimate exclusion); (4) state in two sentences what the curve shows and what it does not. Expected evidence: outputs and statement. Rubric: 3 = ten distinct specifications identified, hostile exclusion computed correctly, statement distinguishes direction from size and notes shared assumptions; 2 = one error; 1 = not run; 0 = none.
EX046 — Question bank. Objective: prepare for the defense. Inputs: §8.2. Constraints: one question from each family; answers of at most 80 words. Steps: write the questions and answers for the Triage Rubric Study. Expected evidence: nine question–answer pairs. Rubric: 3 = questions sharp, answers concede, quantify and propose tests where appropriate; 2 = defensive answers; 1 = questions generic; 0 = none.
EX047 — Counterexample paragraph. Objective: present your own counterexamples. Inputs: Lab 8 sequence results; Lab 6 better/same/worse counts; theme 3. Constraints: 150–200 words. Steps: write the paragraph for the discussion chapter. Expected evidence: the paragraph. Rubric: 3 = S2 and the three worse analysts presented with what they show and do not show, hypotheses distinguished from findings, no dismissal; 2 = counterexamples minimized; 1 = listed only; 0 = none.
EX048 — Reproduction report. Objective: act as the independent reader. Inputs: the eight labs of this book. Constraints: use only what is printed in the book. Steps: (1) attempt to reproduce Labs 6–8 from the printed code; (2) record your Python version and any differences; (3) write a half-page reproduction report listing what succeeded, what failed and what the author should document better. Expected evidence: the report. Rubric: 3 = versions recorded, outputs compared line by line, at least two concrete documentation improvements (for example, the file-name dependency between labs; the seed-choice note); 2 = no versions; 1 = not attempted; 0 = none.
EX049 — Corrections table. Objective: respond to examination. Inputs: three examiner requests: (a) "justify the 30-second timing tolerance"; (b) "the noisy-case result should be described as inconclusive, not null"; (c) "add a mixed-effects logistic model". Constraints: you may not collect new data. Steps: complete the corrections table for each, stating whether the claim changes. Expected evidence: the table. Rubric: 3 = (a) justified from the pilot and measurement needs, (b) accepted with wording change, (c) addressed by running the model or explaining how it relates to the estimand, with claim implications; 2 = one request dismissed; 1 = vague; 0 = none.
EX050 — Partner briefing and next study. Objective: share results and plan what follows. Inputs: the whole study. Constraints: a one-page briefing for MRSS and a half-page next-study proposal. Steps: (1) write the briefing with result, scope, cost, dependence on the reference standard and a recommendation; (2) propose the next study, naming its question, design and the rival it addresses that this study could not. Expected evidence: the two documents. Rubric: 3 = briefing accurate and scoped with no individual data, recommendation conditional, next study addresses a named limitation (live conditions, a second reference panel, rubric-completion timing); 2 = overclaims; 1 = generic; 0 = none.
Chapter synthesis
Examiners look for claims larger than evidence, untraceable numbers and unaddressed rivals. Prepare a reproduction package and ask an independent reader to rerun and attack it. Answer "what if" questions with a specification curve of defensible choices, identifying duplicates and shared assumptions. Present your own counterexamples. In the defense, concede valid points, quantify what they change and propose tests. Record corrections with an audit trail and re-check numbers. Share results with participants first, partners with scope and limits, and the profession with materials others can use to replicate.
Continue studying — chapter 08 of 08
← Chapter 07 · Course chapter index · Next course via chapter index →
This is the final chapter of this course; the index links the next course in the full study path.