Chapter 8 — Independent review and defense
Lesson map
Lesson 08 published. Map, example, EX045.
flowchart TD
N0["Learning outcomes and connection"]
N1["8.1 How examiners read"]
N2["8.2 Families of questions"]
N3["8.3 What if you had chosen differently? — the specif"]
N4["8.4 Preparing for the defense"]
N5["8.5 After the defense: corrections"]
N6["8.6 Sharing results responsibly"]
N7["8.8 Next steps after the master's"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX045"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["8.1 How examiners read"]
N2["8.2 Families of questions"]
N3["8.3 What if you had chosen differently? — the specif"]
N4["8.4 Preparing for the defense"]
N5["8.5 After the defense: corrections"]
N6["8.6 Sharing results responsibly"]
N7["8.8 Next steps after the master's"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX045"]
[Learning outcomes and connection]
|
v
[8.1 How examiners read]
|
v
[8.2 Families of questions]
|
v
[8.3 What if you had chosen differently? — the specif]
|
v
[8.4 Preparing for the defense]
|
v
[8.5 After the defense: corrections]
|
v
[8.6 Sharing results responsibly]
|
v
[8.8 Next steps after the master's]
|
v
[Worked example] --> [Exercise EX045]
Learning outcomes and connection
By the end of this chapter the learner will be able to:
- Explain how examiners typically read a dissertation and prepare the document and the candidate for that reading.
- Anticipate the families of questions examiners ask of an applied study and prepare evidence-based answers, including honest "I don't know" answers.
- Answer "what if you had chosen differently?" with a specification curve that shows how the main estimate depends on defensible analysis choices — and explain why robustness to choices is not the same as accuracy.
- Conduct a mock defense and use it to find gaps in the work, not only in the presentation.
- Respond to an examiners' report with a corrections table that maps every comment to an action and a location.
- Share findings with partners and the sector without disclosing sensitive information, and route any security weakness found incidentally to the right party.
This chapter closes the dissertation cycle that began in Chapter 1. It draws on every earlier chapter: the question and scope (1), the literature and seminar map (2), the design and ethics (3), the rival table (4), the proposal's fixed elements (5), the analysis (6) and the results-to-claims register (7).
8.1 How examiners read
Examination formats differ widely. Some master's dissertations are assessed only from the written document by one internal and one external examiner; others include an oral defense (viva voce) or a presentation. Outcomes commonly include acceptance, acceptance with minor or major corrections, resubmission or failure; your institution's regulations are the only authority on which apply.
Research on how experienced examiners assess theses — mostly at doctoral level but instructive for master's work — suggests patterns worth preparing for. Mullins and Kiley (2002) interviewed experienced examiners and reported that they generally expect a thesis to pass, form early impressions from the opening and closing chapters, look for coherence between the question, the work and the conclusions, and are irritated by carelessness in presentation, which colours their reading of the rest. The practical implications are simple: the abstract, introduction and conclusion must be precise and consistent; the questions must be answered visibly in the order asked; and small errors — inconsistent numbers, broken references, unlabelled figures — cost more than their size suggests. Chapter 7's provenance check and results-to-claims register exist for exactly this reader.
8.2 Families of questions
Examiners' questions on an applied infrastructure study fall into recognizable families. Prepare at least two questions per family, with answers that cite your evidence.
| Family | Typical question | What a good answer contains |
|---|---|---|
| Question and contribution | "Why does this question matter, and what does your study add?" | The decision it informs; the gap from the concept matrix; what is new (controlled design, distance gradient, mechanism) |
| Design choice | "Why not a synthetic control?" | Donor pool too small (two comparison EDs); assumptions of each design compared |
| Data and measurement | "What does an 'ambulance arrival' mean in these systems?" | Custodian definitions; audit; downtime-day rule; what the count misses (acuity) |
| Analysis choices | "What if you had used a different comparison or baseline?" | The specification curve (§8.3) |
| Rivals | "Couldn't the season explain this?" | The rival's distinct prediction and the test: Wexmoor, placebo windows, timing |
| Ethics and sensitivity | "Could your results help an attacker?" | Sensitivity tiers; what was withheld and why; regional reporting |
| Limitations | "What is the weakest part of the study?" | The top-ranked directional limitation and its likely effect |
| Transfer | "Would this happen in a city with ten hospitals?" | The mechanism (load transfer limited by peers' spare capacity and routes) predicts smaller per-hospital effects where load spreads; untested |
| Implications | "What should a coalition do differently?" | Only what the evidence supports, with confidence; what was not tested |
| Next study | "If you had another year?" | The study that would most reduce uncertainty: more events pooled, hourly data, acuity |
8.3 "What if you had chosen differently?" — the specification curve
8.3.1 The idea
Every analysis involves choices that are defensible in more than one way: baseline length, comparison units, the length of the event window, whether to exclude holidays. A prespecified primary analysis protects against choosing after seeing results (Chapter 6), but an examiner may reasonably ask whether the conclusion depends on the particular choices made. A specification curve answers by estimating the effect under every combination of reasonable choices and reporting the whole distribution (Simonsohn, Simmons and Nelson, 2020); a multiverse analysis applies the same idea to data-processing choices (Steegen et al., 2016). The prespecified estimate remains primary; the curve shows how fragile or robust it is.
8.3.2 Practical activity: the Spillover specification curve
Save Lab 4 as lab4_data.py and Lab 6 as lab6_its.py first. The lab below estimates Alder Point's effect under 54 specifications: baseline of 90, 120 or 180 days; comparison W1, W2 or pooled; event window of 14, 21 or 28 days; holidays kept or dropped.
# Lab 8 - specification curve for the examiner's "what if you had chosen differently?" (offline)
# Save Lab 4 as lab4_data.py and Lab 6 as lab6_its.py in the same folder first.
import itertools, statistics
from lab4_data import make_region, DAYS
from lab6_its import cits
d = make_region()
controls = {"W1": d["W1"], "W2": d["W2"], "pooled": {t: d["W1"][t] + d["W2"][t] for t in DAYS}}
specs = []
for pre_len, ctrl, ev_len, drop_hol in itertools.product((90, 120, 180), controls, (14, 21, 28), (False, True)):
drop = frozenset({-100, 45}) if drop_hol else frozenset()
est, lo, hi = cits(d["AP"], controls[ctrl], pre_len=pre_len, ev_len=ev_len, drop=drop)
specs.append((est, lo, hi, pre_len, ctrl, ev_len, drop_hol))
specs.sort()
ests = [s[0] for s in specs]
print(f"specifications: {len(specs)}")
print(f"median effect {statistics.median(ests):+.3f}; range {ests[0]:+.3f} to {ests[-1]:+.3f}")
print(f"intervals excluding zero: {sum(s[1] > 0 for s in specs)} of {len(specs)}")
for label, idx in (("pre_len", 3), ("control", 4), ("event_len", 5), ("drop_holidays", 6)):
groups = {}
for s in specs:
groups.setdefault(s[idx], []).append(s[0])
print(label + ": " + ", ".join(f"{k}={statistics.median(v):+.3f}" for k, v in sorted(groups.items(), key=lambda kv: str(kv[0]))))
print("lowest three:", [(round(s[0], 3), s[3], s[4], s[5], s[6]) for s in specs[:3]])
Expected output:
specifications: 54
median effect +0.245; range +0.199 to +0.332
intervals excluding zero: 54 of 54
pre_len: 120=+0.237, 180=+0.205, 90=+0.271
control: W1=+0.221, W2=+0.259, pooled=+0.238
event_len: 14=+0.237, 21=+0.260, 28=+0.217
drop_holidays: False=+0.244, True=+0.246
lowest three: [(0.199, 180, 'pooled', 14, False), (0.201, 180, 'pooled', 14, True), (0.202, 180, 'W2', 28, False)]
Reading the curve:
- Alder Point's conclusion is robust to these choices. All 54 intervals exclude zero; the median estimate is +24.5% and the range +19.9% to +33.2%. No reasonable choice in this set turns the increase into "no effect".
- Choices still move the estimate. Shorter baselines give larger estimates (median +27.1% with 90 days, +20.5% with 180), because the linear trend fitted to a shorter period is less stable. W2 as the only comparison gives larger estimates than W1. A 21-day window gives the largest median because it matches the true diversion period; 14 days omits part of it and 28 days dilutes it with the tail.
- Holidays barely matter (median +24.4% kept, +24.6% dropped): the regional holidays fall outside the event window and affect both regions.
Running the same curve for Brookfield (EX045) gives a median of +2.5%, a range of −1.5% to +6.9%, and no interval excluding zero. The Brookfield result is robustly inconclusive — yet the generator's true effect was +10%, above every one of the 54 point estimates. This is the specification curve's limit: it shows how much a conclusion depends on analysis choices, but every specification uses the same data with the same noise. Robustness to choices is not accuracy. Only more information — more events, more neighbours, finer data — can resolve Brookfield.
8.4 Preparing for the defense
8.4.1 The opening summary
If the format includes an oral defense, you will usually be asked to summarize the work in 10–15 minutes. Use the structure of the seminar presentation (Chapter 2) applied to your own work: question and why it matters; design and data; main results with intervals; what the design rules out and leaves open; limitations and what next. Show the specification curve if the examiners are likely to ask about choices. End with the scoped answer to each research question in the words of the dissertation.
8.4.2 Mock defenses
Ask two people to examine you for an hour — ideally one methodologist and one practitioner from the sector — with the question families of §8.2. Record the session with their consent. Afterwards, sort every question you answered poorly into one of three kinds:
- Presentation gaps: you knew the answer but could not find it or say it clearly. Fix with preparation.
- Document gaps: the dissertation does not contain information it should. Fix the dissertation before submission if possible, or prepare an honest answer.
- Work gaps: the study cannot answer the question. Do not hide these; prepare to state them as limitations and to describe the study that would address them.
8.4.3 Answering
Use the three registers from the seminar: "the dissertation shows…", "I infer…", "I don't know; it would take…". Concede what should be conceded promptly; defending an indefensible point damages the credibility of the defensible ones. When an examiner proposes an alternative analysis, do not improvise a new result in the room; explain what you would expect and why, and offer to run it as a correction if requested. When asked about the real world, remember the scope: in this book's synthetic study, nothing is known about real hospitals.
8.5 After the defense: corrections
Examiners' reports list required corrections, sometimes with recommendations. Respond with a corrections table:
| # | Examiner comment | Response | Action | Location |
|---|---|---|---|---|
| 1 | "The abstract describes the Brookfield result as showing no effect." | Agreed; the result is inconclusive. | Abstract and §5.2 rewritten with the interval and what it excludes. | Abstract; p. 54 |
| 2 | "Justify ten interviews." | Agreed that justification was thin. | Added an information-power paragraph with the aim, sample specificity and dialogue quality. | §3.6 |
| 3 | "Run the analysis with Calloway as an additional comparison." | Partly agreed: Calloway is part of the distance hypothesis, so using it as a comparison assumes the answer for RQ2. | Added as a sensitivity analysis for RQ1 only, with this caveat. | Appendix F |
| 4 | "The waiting-time result should not appear in the conclusion." | Agreed. | Removed from the conclusion; retained in results with the confounding caveat. | §6 |
Where you disagree, say so politely with reasons and evidence, as in row 3; examiners generally accept a reasoned partial agreement. Keep the table with the corrected dissertation.
8.6 Sharing results responsibly
8.6.1 Partners first
Before any public presentation, give partners the factual and protected-information review promised in the data agreement (Chapter 3), within its fixed period and without a veto on findings. Offer each partner a private briefing that includes anything specific to them — for example, that their ED learned of the diversion informally and late. Findings about a specific organization's weaknesses belong to that organization first.
8.6.2 The sector and the public
A briefing for the regional healthcare coalition can present the regional pattern, the mechanism and the recommendations with confidence levels:
- Moderate confidence: the nearest open ED on the main ambulance routes absorbs most diverted load in the first days; plans should name it and provide surge support.
- Moderate confidence: neighbours need formal, early notification; informal notice through arriving crews cost hours.
- Low confidence (not tested): explicit EMS load-sharing rules would reduce the nearest ED's peak burden.
Do not publish facility-level spare-capacity estimates or a ranked list of which hospital's loss would cause the most regional strain; such outputs would help someone seeking to maximize harm and add little defensive value beyond the organizations concerned (the doctoral book's dual-use questions apply). Synthetic or anonymized regions are acceptable for publication when the structure cannot be matched to a real place.
8.6.3 Incidental security findings
Research on operations sometimes reveals a security weakness — an interviewee mentions an exposed remote-access service, or a data extract contains credentials. Do not investigate it, test it or describe it publicly. Report it privately to the organization concerned through the contact named in your agreement; if it concerns a vendor product rather than one organization's configuration, coordinated vulnerability disclosure through the vendor or a coordinator such as CISA's CVD programme is the established route. Record what you did, and tell your supervisor and the ethics committee if your protocol requires it.
8.7 Worked example: a mock-defense exchange (synthetic)
Examiner (methodologist): "Your main result depends on Wexmoor having the same season. How do you know it did?"
Candidate: "The dissertation shows the pre-event ratio of Alder Point to Wexmoor was flat over 180 days, with no trend break, and the result holds against each Wexmoor hospital separately — the specification curve's lowest estimate is about 20%. I infer that a different season in Wexmoor is unlikely to explain the whole increase. I don't know whether Wexmoor's respiratory season started on exactly the same day; that would take regional surveillance data I did not have, and it is in the limitations with its likely direction: if Wexmoor's season were milder, my estimate would be too high."
Examiner (practitioner): "Brookfield's managers said they were swamped. Your numbers say nothing happened. Who is right?"
Candidate: "My numbers don't say nothing happened — Brookfield's estimate is +5.8% with an interval from about −7% to +20%, so they can't distinguish no change from a substantial one. The interviews describe heavy nights and a concurrent staffing shortage. The joint display reports this as a discordance with three possible explanations. Settling it would need hourly data and staffing records."
Examiner: "Then should coalitions plan for spillover beyond the nearest hospital?"
Candidate: "This study can't tell them how much. It can tell them the nearest ED on the main routes took most of the load in this event, and that the evidence for more distant hospitals is inconclusive rather than negative."
The answers use the three registers, concede promptly, cite evidence and keep the claims inside the design.
8.8 Next steps after the master's
A completed dissertation leaves a list of studies that would reduce its remaining uncertainty: pooling several diversion events with methods for staggered timing; hourly and acuity-weighted outcomes; evaluating a load-sharing protocol. Some of these are practical projects for an operator or coalition; others are doctoral in scale. The doctoral course in this area shows how such a question becomes an original contribution with independent validation. In the study path, the next course is the doctoral Cybersecurity track.
8.9 Common errors and limits
- Treating the abstract and conclusion as formalities; examiners read them first.
- Preparing answers about presentation but not about the work's real gaps.
- Treating a specification curve as proof of truth rather than of robustness to choices.
- Improvising new analyses in the defense.
- Defending every point equally, including indefensible ones.
- Responding to corrections without a table that maps each comment to a change.
- Presenting findings publicly before partners have had their agreed review.
- Publishing facility-level capacity or vulnerability details.
- Investigating an incidental security finding instead of reporting it privately.
The limit of examination is that it judges the dissertation, not the world. Passing confirms that the study was done competently and its claims are bounded; it does not make the synthetic or single-region findings general.
Exercises — chapter 8
EX045 — Specification curves for two sites. Objective: use Lab 8. Inputs: Labs 4, 6 and 8. Constraints: offline. Steps: (1) run Lab 8 and record the output; (2) replace d["AP"] with d["BF"] and record the output; (3) compare the two curves; (4) explain why Brookfield's true 10% effect lies above every point estimate and what this says about specification curves. Expected evidence: outputs and a paragraph. Rubric: 3 = both outputs correct, Brookfield described as robustly inconclusive, the limit (same data, same noise; robustness is not accuracy) explained; 2 = one output wrong; 1 = Brookfield described as "no effect"; 0 = none.
EX046 — Question bank. Objective: prepare for examiners. Inputs: §8.2 and your own or the Spillover dissertation. Constraints: two questions per family; answers of at most 120 words citing evidence. Steps: (1) write the questions; (2) write answers using the three registers; (3) mark the three hardest and the gap each reveals (presentation, document or work). Expected evidence: question bank. Rubric: 3 = hardest questions genuinely hard (parallel trends, Brookfield discordance, transfer), answers cite evidence and concede where appropriate, gaps correctly typed; 2 = answers generic; 1 = questions easy; 0 = none.
EX047 — Corrections table. Objective: respond to examiners. Inputs: a synthetic report with four comments: (a) "The literature review is author-by-author"; (b) "Why was the post-diversion step included? It looks like a way to absorb inconvenient data"; (c) "The ethics section does not mention partner review"; (d) "Please add patient outcomes". Constraints: the table format of §8.5. Steps: respond to each with agree, partly agree or disagree, giving reasons and actions. Expected evidence: table. Rubric: 3 = (a) agreed and restructured by concept, (b) defended with the prespecification and the reason (avoiding contamination of the trend by the tail) and a sensitivity analysis offered, (c) agreed and added, (d) declined as outside scope and approvals with a future-work note; 2 = one response unreasoned; 1 = agrees to everything including (d); 0 = none.
EX048 — Mock defense and gap sorting. Objective: use a mock defense to improve the work. Inputs: your dissertation or the Spillover materials. Constraints: a 45-minute mock with two examiners; consented recording. Steps: (1) run the mock; (2) list every weak answer; (3) sort them into presentation, document and work gaps; (4) fix what can be fixed and draft limitation text for the rest. Expected evidence: list, sorting and fixes. Rubric: 3 = honest sorting with at least one work gap acknowledged, document fixes made, limitation text directional; 2 = all gaps classed as presentation; 1 = no mock; 0 = none.
EX049 — Coalition briefing. Objective: share findings responsibly. Inputs: the Spillover results and themes. Constraints: one page; recommendations with confidence levels; a "what this study does not show" paragraph; no facility-level capacity figures. Steps: (1) draft; (2) mark each sentence with its source in the dissertation; (3) list what was shared privately with individual partners instead. Expected evidence: briefing and list. Rubric: 3 = recommendations traceable and calibrated, untested recommendations labelled, limits explicit, sensitive details routed privately; 2 = one uncalibrated recommendation; 1 = facility-specific weaknesses published; 0 = none.
EX050 — The next study. Objective: turn limitations into a feasible proposal. Inputs: the directional limitations in §7.6. Constraints: one page; choose the limitation whose resolution would most change the conclusions. Steps: (1) justify the choice; (2) state the question and estimand; (3) choose a design and the data it needs; (4) state the result that would change planners' decisions; (5) say whether it is a master's, doctoral or practitioner project and why. Expected evidence: one-page proposal. Rubric: 3 = chosen limitation is consequential (for example a single event), design feasible (pooled events with staggered-timing methods, or hourly acuity-weighted data), decision-relevant result stated, scale judgement reasoned; 2 = design vague; 1 = repeats the original study; 0 = none.
Chapter synthesis
Examiners read for coherence and are influenced by care; make the abstract, introduction and conclusion precise and consistent, and the answers to each question visible. Prepare for predictable families of questions with evidence. Answer "what if" with a specification curve that keeps the prespecified estimate primary and shows its dependence on reasonable choices — remembering that robustness to choices is not accuracy, as Brookfield's robustly inconclusive estimate of a real effect shows. Use mock defenses to find gaps in the work, not only in the talk. Respond to corrections with a mapped table and reasons. Share with partners first, publish regional patterns rather than facility-level weaknesses, and route incidental security findings privately. The dissertation's limitations become the next study.
Continue studying — chapter 08 of 08
← Chapter 07 · Course chapter index · Next course via chapter index →
This is the final chapter of this course; the index links the next course in the full study path.