Chapter 7 — Dissertation on response effectiveness and limits

Lesson map

Lesson 07 published. Map, example, EX035.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["7.1 Structure: one job per chapter"]
  N2["7.2 The argument and the results-to-claims register"]
  N3["7.3 Reporting results honestly"]
  N4["7.4 Number provenance"]
  N5["7.5 Practical activity: number provenance"]
  N6["7.6 Reporting guidelines as checklists"]
  N7["7.7 Directional limitations"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX035"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["7.1 Structure: one job per chapter"]
  N2["7.2 The argument and the results-to-claims register"]
  N3["7.3 Reporting results honestly"]
  N4["7.4 Number provenance"]
  N5["7.5 Practical activity: number provenance"]
  N6["7.6 Reporting guidelines as checklists"]
  N7["7.7 Directional limitations"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX035"]

[Learning outcomes and connection]
    |
    v
[7.1 Structure: one job per chapter]
    |
    v
[7.2 The argument and the results-to-claims register]
    |
    v
[7.3 Reporting results honestly]
    |
    v
[7.4 Number provenance]
    |
    v
[7.5 Practical activity: number provenance]
    |
    v
[7.6 Reporting guidelines as checklists]
    |
    v
[7.7 Directional limitations]
    |
    v
[Worked example] --> [Exercise EX035]

Learning outcomes and connection

After this chapter you should be able to:

1. Structure a dissertation so that each chapter has one job, and write an argument that a reader can follow from question to conclusion.
2. Build a results-to-claims register that keeps every claim inside the evidence that supports it.
3. Report estimates with intervals and calibrated language, including negative and inconclusive results.
4. Check number provenance automatically so that every number in the text matches the analysis output.
5. Use reporting guidelines as checklists, state limitations with their direction, and write a data and code availability statement under confidentiality constraints.

The chapter builds on the results of Chapter 6 and prepares Chapter 8, where the dissertation faces independent review.

7.1 Structure: one job per chapter

A conventional empirical dissertation has a recognizable shape, and examiners read faster when you use it:

Lesson table; scroll horizontally to read all columns →

The single most useful discipline is to keep results and interpretation apart. The results chapter says what the numbers and themes are; the discussion says what they mean. When a sentence in the results chapter contains "suggests", "demonstrates" or "because", it probably belongs in the discussion.

7.2 The argument and the results-to-claims register

A dissertation's argument is a chain: the problem matters; the question addresses it; the design can answer the question; the analysis was sound; the results are these; therefore these claims follow, within these limits. A results-to-claims register makes each link inspectable:

Lesson table; scroll horizontally to read all columns →

The register is a working document, but including a condensed version in the discussion chapter shows examiners you have thought about claim strength.

7.3 Reporting results honestly

7.3.1 Estimates and intervals, not verdicts

Report the estimate, its interval and the test, in that order of importance: "the mean within-analyst difference was −17.4 percentage points (95% bootstrap interval −24.5 to −9.8; sign-flip permutation p = 0.0005)". Avoid "significant" as a description of importance; if you use it, say "statistically significant at the 5% level" and give the interval anyway. Never write "the rubric works" or "proves".

7.3.2 Calibrated verbs

Lesson table; scroll horizontally to read all columns →

7.3.3 Negative and inconclusive results

A negative result — an interval centred on zero and excluding effects of practical interest — is a finding: "the data are inconsistent with a reduction larger than X". An inconclusive result — an interval including both no effect and important effects — is not a negative result and must not be reported as "no effect". In this study, the over-triage comparison and the noisy-case subgroup are inconclusive in the strict sense; the dissertation says "no increase was observed" and "no effect was detectable on noisy cases, which the design was not powered to rule out", not "the rubric has no effect on noisy cases".

7.3.4 Multiplicity

The study has one primary outcome, four secondary outcomes, a subgroup analysis and four robustness checks. Only the primary outcome carries the confirmatory weight; the others are reported with their intervals and described as secondary or exploratory. If you report many p-values, say how many tests were done, so a reader can judge the chance of a false positive among them.

7.4 Number provenance

Most numerical errors in dissertations are not analysis errors; they are transcription errors — a number from an earlier run, a typo, a rounding inconsistency, a sample size that was not updated after a withdrawal. The remedy is mechanical: every number in the text comes from an output file, and before submission a script checks that each printed number matches. Some students generate tables and in-text numbers directly from code; others keep a list of printed numbers with their source keys, as in Lab 7. Either is acceptable; copying by hand is not.

7.5 Practical activity: number provenance

Lab 7 compares nine numbers printed in a synthetic draft dissertation with the analysis output. Save it as lab7_provenance.py next to lab6_data.py and lab6_analysis.py.

# Lab 7 - number provenance: does every number in the draft dissertation match the analysis output? (offline)
# Imports lab6_data.py and lab6_analysis.py (save both). The draft sentences below are synthetic.
import statistics as st
from lab6_analysis import ROWS, paired, boot_ci, signflip_p, wkappa
d, R, U, both = paired(ROWS)
lo, hi = boot_ci(d)
med = {c: st.median(r["seconds"] for r in ROWS if r["cond"] == c and r["analyst"] in both) for c in "UR"}
kap = {c: wkappa([(r["ref"], r["rating"]) for r in ROWS if r["cond"] == c and r["analyst"] in both]) for c in "UR"}
RESULTS = {  # key -> (value, how to render it for comparison with the text)
    "n_complete": (len(both), "{:d}"),
    "diff_pp": (100 * st.mean(d), "{:.1f}"),
    "ci_lo_pp": (100 * lo, "{:.1f}"), "ci_hi_pp": (100 * hi, "{:.1f}"),
    "p_signflip": (signflip_p(d), "{:.4f}"),
    "kappa_R": (kap["R"], "{:.2f}"), "kappa_U": (kap["U"], "{:.2f}"),
    "extra_seconds": (med["R"] - med["U"], "{:.0f}"),
}
DRAFT = [  # (location, number as printed, result key)
    ("Abstract", "23", "n_complete"),
    ("Abstract", "-17.4", "diff_pp"),
    ("Abstract", "-24.5", "ci_lo_pp"),
    ("Abstract", "-9.8", "ci_hi_pp"),
    ("Results 5.2", "0.0005", "p_signflip"),
    ("Results 5.4", "0.78", "kappa_R"),
    ("Results 5.4", "0.62", "kappa_U"),
    ("Results 5.5", "38", "extra_seconds"),
    ("Discussion 6.1", "24", "n_complete"),
]
bad = 0
for where, printed, key in DRAFT:
    value, fmt = RESULTS[key]
    ok = fmt.format(value) == printed
    bad += not ok
    print(f"{'ok  ' if ok else 'FAIL'} {where:<15} printed {printed:>7}  analysis {fmt.format(value):>7}  [{key}]")
print(f"{bad} of {len(DRAFT)} printed numbers do not match the analysis output")

Expected output:

ok   Abstract        printed      23  analysis      23  [n_complete]
ok   Abstract        printed   -17.4  analysis   -17.4  [diff_pp]
ok   Abstract        printed   -24.5  analysis   -24.5  [ci_lo_pp]
ok   Abstract        printed    -9.8  analysis    -9.8  [ci_hi_pp]
ok   Results 5.2     printed  0.0005  analysis  0.0005  [p_signflip]
FAIL Results 5.4     printed    0.78  analysis    0.76  [kappa_R]
ok   Results 5.4     printed    0.62  analysis    0.62  [kappa_U]
FAIL Results 5.5     printed      38  analysis      45  [extra_seconds]
FAIL Discussion 6.1  printed      24  analysis      23  [n_complete]
3 of 9 printed numbers do not match the analysis output

How to read it. Six numbers match. Three do not. The rubric kappa of 0.78 is a stale value from an earlier run before observer form v1.1 changed which rating was recorded; the time cost of 38 seconds was computed from means in an earlier draft and never updated to the predeclared medians (45 seconds); and the discussion's "24 analysts" counts enrolled analysts where the analysis used 23 with both periods. None of these errors changes the conclusion, but each would cost credibility with an examiner and the third misstates the sample. The fix is to regenerate the numbers, not to edit the script's expectations.

7.6 Reporting guidelines as checklists

Reporting guidelines list what a reader needs in order to judge a study. They are not quality scores and they do not dictate design; they are checklists of what to say. The relevant ones here are:

• CONSORT for randomized trials (Schulz, Altman and Moher 2010) and its extension to randomized crossover trials (Dwan et al. 2019), which asks specifically about carryover, period effects and the analysis of within-participant data — directly relevant to the tabletop.
• STROBE for observational studies (Vandenbroucke et al. 2007), for the historical MRSS strand.
• SRQR, Standards for Reporting Qualitative Research (O'Brien et al. 2014), for the debrief interviews.
• CONSORT pilot extension (Eldridge et al. 2016) for the pilot report, if it is presented as a separate study.

Work through each checklist after drafting the methods and results, and record where each item is addressed. Items you cannot address (for example, a registry number if the plan was not registered) should be acknowledged, not ignored.

7.7 Directional limitations

A limitations section that lists weaknesses without saying what they do to the answer is decoration. For each limitation, state its direction — does it probably make the estimate too large, too small, or affect only its precision or transferability? — and, where possible, its likely size.

Lesson table; scroll horizontally to read all columns →

7.8 Accessible figures and tables

Choose figures that show the unit of analysis: a dot plot of the 23 per-analyst differences, with the mean and interval, shows both the effect and its spread better than a bar chart of two pooled proportions. Use colour-blind-safe palettes and do not encode meaning in colour alone; give every figure a text alternative that states its message; label axes with units; and make tables readable by screen readers (no merged cells used for layout). An accessible dissertation is easier for every examiner to read.

7.9 Data and code availability under sensitivity

The availability statement says what is shared, where, and under what conditions. For this study: the analysis code, the pre-analysis plan, the case vignettes, the rubric and the observer form are deposited openly; the de-identified decision-level dataset is available to bona fide researchers under a data use agreement, because combinations of team, experience band and sequence could identify participants; a synthetic dataset with the same structure (produced by lab6_data.py) is deposited openly so that the pipeline can be run; interview transcripts are not shared, as participants were promised, but the codebook and audit trail are. This follows the spirit of the FAIR principles (Wilkinson et al. 2016): metadata and access routes are open even where the data themselves are restricted.

7.10 Worked example: rewriting the abstract (synthetic)

First draft. "This study proves that structured triage rubrics significantly improve SOC performance. Analysts using the SEV-4 rubric were 32% less likely to miss critical incidents (p < 0.001). Organizations should adopt rubrics."

Problems. "Proves" overclaims; "SOC performance" is broader than the outcome; "32% less likely" is a relative reduction (0.174/0.538) presented without the absolute difference; "critical incidents" conflates high-impact (3–4) with critical (4); "p < 0.001" is false (p = 0.0005 satisfies it, but the interval is the main result and is missing); the recommendation ignores scope and cost.

Revised (structured). Background: analysts may under-rate high-impact incidents with little alert noise. Methods: randomized crossover tabletop; 24 experienced volunteer analysts (23 completed both periods) rated 24 fictional cases against an independent panel reference, with and without a four-question impact rubric. Results: under-triage of high-impact cases fell from 53.8% to 36.4% (mean within-analyst difference −17.4 percentage points, 95% interval −24.5 to −9.8). The reduction was concentrated in quiet-impact cases. Median decision time rose by 45 seconds; over-triage of low-impact cases did not increase. Conclusions: in a controlled setting, the rubric reduced under-triage of high-impact cases at a modest time cost. Its effect in live operations, where quiet-impact cases are fewer and interruptions more common, is likely smaller and remains to be tested.

7.11 Common errors and limits

• Mixing results and interpretation.
• Claims that exceed the design (causal language on associational data; prevalence language on enriched cases).
• Relative reductions without absolute differences.
• Reporting an inconclusive result as "no effect".
• Hand-copied numbers that drift from the analysis.
• Limitations without directions.
• Availability statements that say "data available on request" without conditions or a route.

The limit of writing is that it cannot add evidence. It can only represent the evidence accurately — which, for an examiner, is most of what is being assessed.

Exercises — chapter 7

EX039 — Results-to-claims register. Objective: keep claims within evidence. Inputs: Chapter 6 results and §7.2. Constraints: add three rows. Steps: add rows for (a) the kappa result, (b) theme 4 on escalation, (c) a claim about novices. For each, complete all five columns. Expected evidence: the rows. Rubric: 3 = kappa claimed as secondary with scope, theme 4 as a participant account generating a hypothesis, novice claim marked not supported (eligibility required six months' experience); 2 = one overclaim; 1 = columns incomplete; 0 = none.

EX040 — Run and fix Lab 7. Objective: enforce number provenance. Inputs: Lab 7. Constraints: fix the draft, not the analysis. Steps: (1) run it; (2) correct the three draft entries; (3) rerun and confirm 0 failures; (4) add two entries for the over-triage proportions and confirm they pass. Expected evidence: outputs. Rubric: 3 = draft corrected to 0.76, 45 and 23, rerun shows 0 of 9 (then 0 of 11) failures, new entries keyed correctly; 2 = analysis changed instead; 1 = not run; 0 = none.

EX041 — Calibrated sentences. Objective: match verbs to evidence. Inputs: five draft sentences: (a) "The rubric eliminates under-triage of quiet incidents." (b) "Senior analysts are better at triage." (c) "The rubric has no effect on noisy cases." (d) "Analysts hate the rubric." (e) "MRSS should mandate the rubric." Constraints: use §7.3.2. Steps: rewrite each. Expected evidence: five sentences. Rubric: 3 = each rewritten with the right verb, scope and uncertainty; (c) as inconclusive; (e) as a conditional implication; 2 = one overclaim remains; 1 = cosmetic changes; 0 = none.

EX042 — Directional limitations. Objective: state limitations with direction. Inputs: §7.7. Constraints: two new limitations. Steps: add two limitations (for example, case writers knew the hypothesis; the rubric was designed by the student) with direction and reasoning. Expected evidence: two rows. Rubric: 3 = directions argued (both plausibly favour the rubric, through case construction and allegiance), mitigations or checks proposed; 2 = direction asserted without reasoning; 1 = listed without direction; 0 = none.

EX043 — Reporting checklist audit. Objective: use a guideline as a checklist. Inputs: the CONSORT crossover extension. Constraints: read the checklist items yourself. Steps: (1) choose five items specific to crossover trials; (2) say where the dissertation addresses each, or what is missing. Expected evidence: a table. Rubric: 3 = items genuinely crossover-specific (rationale for crossover, carryover, period effects, washout, within-participant analysis), locations or gaps precise; 2 = generic CONSORT items; 1 = fewer than five; 0 = none.

EX044 — Availability statement. Objective: share responsibly. Inputs: §7.9. Constraints: 120–180 words. Steps: write the statement for a version of the study that also used real, de-identified MRSS tickets under a partner agreement prohibiting disclosure. Expected evidence: the statement. Rubric: 3 = each artefact's access level stated, ticket data's restriction and route described, synthetic stand-in offered, no promise the agreement does not allow; 2 = vague route; 1 = "available on request"; 0 = none.

Chapter synthesis

Give each dissertation chapter one job and keep results apart from interpretation. Build a results-to-claims register so every claim has evidence, design support, scope and an honest strength. Report estimates with intervals in calibrated language, distinguish negative from inconclusive results, and keep the primary outcome's confirmatory role clear. Check every printed number against the analysis output mechanically. Use reporting guidelines as checklists, write limitations with their direction, design accessible figures, and state data and code availability precisely under the constraints you accepted.