Chapter 4 — Competing explanations of response outcomes

Chapter 4 — Competing explanations of response outcomes

Lesson map

Lesson 04 published. Map, example, EX021.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["4.1 Multiple working hypotheses and strong inference"]
  N2["4.2 The historical puzzle at MRSS synthetic"]
  N3["4.3 A rival table"]
  N4["4.4 Causal diagrams"]
  N5["4.5 Discriminating tools and their assumptions"]
  N6["4.6 Competing explanations in a documented study"]
  N7["4.8 Practical activity: the rubric, the senior hires"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX021"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["4.1 Multiple working hypotheses and strong inference"]
  N2["4.2 The historical puzzle at MRSS synthetic"]
  N3["4.3 A rival table"]
  N4["4.4 Causal diagrams"]
  N5["4.5 Discriminating tools and their assumptions"]
  N6["4.6 Competing explanations in a documented study"]
  N7["4.8 Practical activity: the rubric, the senior hires"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX021"]

[Learning outcomes and connection]
    |
    v
[4.1 Multiple working hypotheses and strong inference]
    |
    v
[4.2 The historical puzzle at MRSS synthetic]
    |
    v
[4.3 A rival table]
    |
    v
[4.4 Causal diagrams]
    |
    v
[4.5 Discriminating tools and their assumptions]
    |
    v
[4.6 Competing explanations in a documented study]
    |
    v
[4.8 Practical activity: the rubric, the senior hires]
    |
    v
[Worked example] --> [Exercise EX021]

Learning outcomes and connection

After this chapter you should be able to:

1. Apply the method of multiple working hypotheses to an incident-response outcome and write a rival table whose rows make different predictions.
2. Draw a causal diagram for a historical response dataset and identify confounders, mediators and variables you must not adjust for.
3. Use difference-in-differences, stratification and standardization to challenge a favoured claim, stating the assumption each relies on.
4. Design negative controls and sensitivity analyses that would embarrass the favoured explanation if it were wrong.
5. Explain why the historical strand of the Triage Rubric Study is reported as context, while the causal test is the controlled tabletop.

The chapter builds on the design choice of Chapter 3 and prepares Chapter 5, where the rival table becomes part of the proposal.

4.1 Multiple working hypotheses and strong inference

In 1890 the geologist T. C. Chamberlin warned that a researcher with a single favoured hypothesis comes to see only the evidence that supports it, and recommended holding several working hypotheses at once (Chamberlin, reprinted 1965). Platt (1964) turned the idea into strong inference: devise alternative hypotheses, devise an observation whose possible outcomes exclude one or more of them, carry it out, and repeat. The value of a rival is not that it is likely; it is that it predicts something different from your hypothesis, so that an observation can tell them apart.

In incident response, outcomes have many causes besides the practice being studied. When under-triage falls after a rubric is introduced, the candidates include the rubric, but also new staff, a change in the kinds of incidents arriving, a change in telemetry that makes impact more visible, a change in how severity is recorded, regression from an unusually bad period, attention from being studied, and plain chance. Your job is to list them, say what each predicts, and find data that discriminate.

4.2 The historical puzzle at MRSS (synthetic)

MRSS's East team adopted SEV-4 on 1 January 2025. Its under-triage of high-impact tickets — judged against a panel review of a random sample of tickets after closure — fell. West did not adopt the rubric. The MRSS manager's slide reads: "East under-triage 9% vs West 18%: the rubric halves missed incidents."

What else happened? In January 2025 East also absorbed three senior analysts from a closed contract, raising its share of senior analysts from about a third to about two-thirds. East also handles a different mix of incidents: more school-network alerts with noisy signals, fewer of the quiet-impact incidents (a scheduled job silently failing at the water authority, a payroll export copied to an unusual destination) that are easiest to under-rate.

4.3 A rival table

Two features make this table useful. Each row predicts something specific, and the last column names data that the study can actually obtain. A rival table that says only "confounding may exist" is not a rival table.

4.4 Causal diagrams

A causal diagram (a directed acyclic graph) draws variables as nodes and assumed causal effects as arrows. It is a compact way to state assumptions and to decide what to adjust for. Tools such as dagitty (Textor et al. 2017) check diagrams formally; for a master's project a careful hand drawing is enough.

                 Team (East/West) ──────────────┐
                    │          │                 │
                    v          v                 v
   Contract closure → Seniority  Case mix (quiet share)   Rubric adoption
                    │          │                 │
                    └────┬─────┴─────────────────┘
                         v
                  Under-triage  <── Severity recording practice
                         │
                         v
                  Panel-reviewed outcome

Reading the diagram: seniority and case mix both affect under-triage and are associated with the team, and therefore with rubric adoption; they are confounders of the rubric–outcome relationship and must be accounted for. Recording practice affects the measured outcome; if it changed with adoption it is a confounder too, which is why the outcome is panel review rather than the ticket's own field. A variable that the rubric itself changes on the way to the outcome — for example, "time spent reading the asset inventory" — is a mediator; adjusting for it would remove part of the effect you want to estimate. A variable caused by both the rubric and the outcome (for example, "ticket later reopened") is a collider; adjusting for it can create spurious associations. The diagram is an assumption, not a finding: say where you are unsure.

4.5 Discriminating tools and their assumptions

• Difference-in-differences (DiD) compares the change in East with the change in West over the same period. It removes stable differences between teams and shocks common to both. It assumes parallel trends: without the rubric, East would have changed as West did. A co-intervention specific to East — the senior hires — violates that assumption.
• Stratification and standardization compute outcomes within strata of a confounder (senior/junior × quiet/noisy) and then reweight them to a common reference mix. They remove confounding by the measured variables, and only those. With many strata and few tickets per stratum, estimates become unstable.
• Negative-control outcomes are outcomes the exposure cannot plausibly affect but that share the confounders. If the rubric cannot affect routing errors but seniority can, a "rubric effect" on routing is evidence of confounding. A negative control does not prove the absence of bias when null, but a non-null one is a clear warning.
• Sensitivity analysis asks how strong an unmeasured confounder would have to be to explain away the estimate. At master's level a simple version is enough: "if the senior hires reduced under-triage by as much as seniority does in West, how much of East's DiD would remain?"
• Simpson's paradox — an association that reverses or vanishes when data are split by a third variable (Simpson 1951) — is the extreme case of confounding by mix. It is a reminder that a crude comparison can point the wrong way.

4.6 Competing explanations in a documented study

The surgical safety checklist study of Haynes et al. (2009) reported lower complication and death rates after a checklist was introduced in eight hospitals. Its design — before and after, without a concurrent control group — is common in operational improvement work, and the rivals a reader must consider are exactly those in §4.3: secular improvement, changes in the case mix or recording, and the effect of being observed. The point here is not to judge that study, which later research has extended and debated; it is that the same reasoning applies whenever a security team reports "after adopting the playbook, metric X improved". The incident-response literature has few controlled studies, so the habit of asking "what else changed?" is a core research skill.

4.7 Worked example: building the rival table into the proposal (synthetic)

The student's first draft proposal said: "Historical data from MRSS show that the rubric halved under-triage." Her supervisor asked for the rival table. Working through it changed three things.

First, the historical strand was demoted: its role became to describe the problem (under-triage of quiet-impact incidents occurs and matters) and to test whether the historical data are consistent with a rubric effect after accounting for seniority and case mix — not to estimate the effect. Second, the controlled tabletop became the primary causal test, because its design breaks the link between rubric use and seniority (each analyst uses both conditions) and between rubric use and case mix (case sets are matched and counterbalanced). Third, the tabletop inherited a prediction from the rival table: if the rubric works by drawing attention to impact, its effect should be concentrated in quiet-impact cases. That subgroup analysis was predeclared.

4.8 Practical activity: the rubric, the senior hires and the case mix

Lab 4 generates synthetic panel-reviewed high-impact tickets for East and West in 2024 and 2025 (1,500 per team per year). The generator encodes a true rubric effect of −3 percentage points in East in 2025, a seniority effect of −10 points, a quiet-impact effect of +22 points and a small improvement in both teams in 2025. Routing errors depend on seniority but not on the rubric. Save it as lab4_rivals.py.

# Lab 4 - competing explanations for a fall in under-triage after a rubric was adopted (offline, stdlib only)
# Synthetic historical tickets from two fictional teams at Marlow Regional Shared Services (MRSS).
# East adopted the SEV-4 rubric on 2025-01-01 and, in the same month, gained senior analysts; West changed nothing.
# Only high-impact tickets (panel-reviewed reference severity 3-4) are generated; outcome = under-triaged (0/1).
import random
def make_tickets(seed=404, rubric_effect=-0.03):
    rng = random.Random(seed); rows = []
    for team in ("East", "West"):
        for year in (2024, 2025):
            n = 1500
            senior_share = {("East", 2024): 0.35, ("East", 2025): 0.65,
                            ("West", 2024): 0.35, ("West", 2025): 0.38}[(team, year)]
            quiet_share = {"East": 0.40, "West": 0.55}[team]          # quiet-impact cases are harder
            for _ in range(n):
                senior = rng.random() < senior_share
                quiet = rng.random() < quiet_share
                p = 0.10 + (0.22 if quiet else 0) + (-0.10 if senior else 0) + (-0.02 if year == 2025 else 0)
                if team == "East" and year == 2025:
                    p += rubric_effect
                under = rng.random() < max(p, 0.01)
                misrouted = rng.random() < (0.08 if senior else 0.14)   # rubric does not touch routing
                rows.append(dict(team=team, year=year, senior=senior, quiet=quiet, under=under, misrouted=misrouted))
    return rows
def rate(rows, key="under", **f):
    s = [r for r in rows if all(r[k] == v for k, v in f.items())]
    return sum(r[key] for r in s) / len(s), len(s)
if __name__ == "__main__":
    T = make_tickets()
    e25, _ = rate(T, team="East", year=2025); w25, _ = rate(T, team="West", year=2025)
    e24, _ = rate(T, team="East", year=2024); w24, _ = rate(T, team="West", year=2024)
    print(f"A crude 2025 contrast   East {e25:.3f}  West {w25:.3f}  difference {e25 - w25:+.3f}")
    print(f"B East before/after     2024 {e24:.3f}  2025 {e25:.3f}  change {e25 - e24:+.3f}")
    print(f"C difference-in-diff    East change {e25 - e24:+.3f} minus West change {w25 - w24:+.3f} = {(e25 - e24) - (w25 - w24):+.3f}")
    # D: stratify by seniority and case type, then standardize to the pooled 2024 mix (both teams)
    base = [r for r in T if r["year"] == 2024]
    strata = [(s, q) for s in (False, True) for q in (False, True)]
    w = {k: sum(1 for r in base if (r["senior"], r["quiet"]) == k) / len(base) for k in strata}
    def std(team, year):
        return sum(w[k] * rate(T, team=team, year=year, senior=k[0], quiet=k[1])[0] for k in strata)
    sd = (std("East", 2025) - std("East", 2024)) - (std("West", 2025) - std("West", 2024))
    print(f"D standardized DiD      (seniority x case type held at the 2024 mix) = {sd:+.3f}")
    em25, _ = rate(T, "misrouted", team="East", year=2025); em24, _ = rate(T, "misrouted", team="East", year=2024)
    wm25, _ = rate(T, "misrouted", team="West", year=2025); wm24, _ = rate(T, "misrouted", team="West", year=2024)
    print(f"E negative control      misrouting DiD = {(em25 - em24) - (wm25 - wm24):+.3f}  (rubric cannot affect routing)")
    print(f"  senior share          East 2024 {rate(T, 'senior', team='East', year=2024)[0]:.2f} -> 2025 {rate(T, 'senior', team='East', year=2025)[0]:.2f};"
          f"  West {rate(T, 'senior', team='West', year=2024)[0]:.2f} -> {rate(T, 'senior', team='West', year=2025)[0]:.2f}")
    print(f"  quiet-impact share    East {rate(T, 'quiet', team='East')[0]:.2f}  West {rate(T, 'quiet', team='West')[0]:.2f}")
    print(f"  generator truth       rubric effect on East 2025 = -0.030 (absolute risk)")

Expected output:

A crude 2025 contrast   East 0.093  West 0.179  difference -0.086
B East before/after     2024 0.159  2025 0.093  change -0.066
C difference-in-diff    East change -0.066 minus West change -0.013 = -0.053
D standardized DiD      (seniority x case type held at the 2024 mix) = -0.038
E negative control      misrouting DiD = -0.017  (rubric cannot affect routing)
  senior share          East 2024 0.34 -> 2025 0.65;  West 0.33 -> 0.38
  quiet-impact share    East 0.42  West 0.55
  generator truth       rubric effect on East 2025 = -0.030 (absolute risk)

How to read it. The manager's crude 2025 comparison (A) says the rubric is associated with an 8.6-point reduction; East's before–after change (B) says 6.6 points. Both mix the rubric with other things. The crude contrast is inflated by case mix: East handles fewer quiet-impact cases (42% versus 55%), so it would have lower under-triage without any rubric. The before–after change includes the improvement common to both teams. Difference-in-differences (C) removes the common change and the stable team difference, giving −5.3 points — still too large, because East's senior hires are an East-specific co-intervention that DiD cannot remove. Standardizing to the 2024 mix of seniority and case type (D) gives −3.8 points, close to the generator's true −3.0. The negative-control outcome (E) is the warning light: misrouting, which the rubric cannot affect, shows a "DiD effect" of −1.7 points in East — exactly what an East-specific seniority change would produce. Note also that even with 1,500 tickets per team per year the remaining gap between D and the truth is within sampling noise; with the few hundred tickets a real panel could review, the DiD would be imprecise as well as biased.

The conclusion the student writes is calibrated: "The historical data are consistent with a modest rubric effect after accounting for seniority and case mix, but they cannot separate the rubric from the simultaneous senior hires with confidence; the negative-control outcome indicates residual confounding. The causal question is therefore addressed by the controlled study."

4.9 The tabletop as the answer to the rival table

Return to the rival table and ask what the crossover tabletop does to each row. R1 (seniority) and R2 (case mix) are broken by design: every analyst does both conditions on matched case sets. R3 (secular trend) becomes a period effect, balanced by order and estimable. R4 (recording) is controlled because the outcome is computed from the analyst's rating against a fixed reference, not from a ticket field. R5 (regression to the mean) does not arise, because nobody was selected for a bad period. R6 (chance) remains and is handled with intervals. New rivals appear that the historical data did not have: carryover, observers' expectations, participants guessing the hypothesis, and the artificiality of fictional cases. The tabletop does not abolish rivals; it trades a set it cannot control for a set it can.

4.10 Common errors and limits

• Listing "confounding" as a limitation instead of naming confounders and what they predict.
• Writing rivals that make the same predictions as the favoured hypothesis, so that no observation could separate them.
• Treating DiD as a cure for confounding when a co-intervention is specific to the treated group.
• Adjusting for mediators or colliders because "more adjustment is safer".
• Using the ticket's own severity field as the outcome when recording practice may have changed.
• Ignoring a non-null negative control because the main result looks good.
• Concluding "no confounding" from a null negative control.

The limit is that observational adjustment handles only what you measured and modelled correctly. That is why a design that removes rivals by construction is worth its artificiality for a causal question — and why its own new rivals must be listed too.

Exercises — chapter 4

EX021 — Rival table for a different claim. Objective: write discriminating rivals. Inputs: the claim "after MRSS moved to 12-hour shifts, mean time to contain fell by 20%". Constraints: at least five rivals; each with a distinct prediction. Steps: complete the four columns of §4.3 for each. Expected evidence: a table. Rubric: 3 = rivals specific (case mix, staffing level, tooling, recording of "contained", censoring, regression), predictions distinct and data obtainable; 2 = one rival non-discriminating; 1 = generic; 0 = none.

EX022 — Causal diagram. Objective: encode assumptions and adjustment decisions. Inputs: §4.4. Constraints: add two variables: "telemetry coverage of the affected service" and "ticket reopened". Steps: (1) redraw the diagram with them; (2) classify every variable as confounder, mediator, collider or other with respect to rubric → under-triage; (3) state the adjustment set. Expected evidence: the diagram and a table. Rubric: 3 = telemetry correctly placed (a confounder if it differs by team and affects triage, or an effect modifier), reopened identified as a collider or post-outcome variable not to adjust for, adjustment set justified; 2 = one misclassification; 1 = diagram without reasoning; 0 = none.

EX023 — Run and modify Lab 4. Objective: see when DiD works. Inputs: Lab 4. Constraints: offline. Steps: (1) run it; (2) change East's 2025 senior share from 0.65 to 0.38 (the same as West's), rerun and report lines A–E; (3) explain why DiD and standardized DiD now agree with each other and with the truth more closely; (4) explain what happened to the negative control. Expected evidence: outputs and a 150-word explanation. Rubric: 3 = outputs correct, the parallel-trends logic explained, negative control's movement towards zero linked to removing the co-intervention; 2 = one error; 1 = not run; 0 = none.

EX024 — Negative controls. Objective: design negative controls for the tabletop. Inputs: the tabletop design of Chapter 3. Constraints: at least two negative controls, one outcome and one exposure. Steps: for each, state what it is, why the rubric cannot affect it (or why it cannot be the cause), and what a non-null result would mean. Expected evidence: a half page. Rubric: 3 = controls plausible (for example, accuracy on a set of identification questions unrelated to severity; a "sham" condition with an unrelated checklist), interpretation of non-null results specific; 2 = one control implausible; 1 = definitions only; 0 = none.

EX025 — Sensitivity statement. Objective: reason about unmeasured confounding. Inputs: Lab 4 output; suppose seniority were unmeasured. Constraints: use only numbers from the lab or the generator description. Steps: (1) estimate how much of DiD (C) the senior hires could explain, given a seniority effect of −10 points and the change in East's senior share relative to West's; (2) state what remains; (3) write a two-sentence sensitivity statement. Expected evidence: the calculation and statement. Rubric: 3 = calculation ≈ −10 × (0.31 − 0.05) ≈ −2.6 points, remainder ≈ −2.7 points, statement calibrated; 2 = arithmetic slip; 1 = no number; 0 = none.

EX026 — Rivals for the tabletop. Objective: list the rivals a controlled design introduces. Inputs: §4.9. Constraints: at least four rivals specific to the tabletop. Steps: for each, give its prediction, a mitigation in the protocol and a check in the analysis. Expected evidence: a table. Rubric: 3 = carryover, observer expectation, demand characteristics, fidelity failure and artificiality covered with concrete checks; 2 = one missing check; 1 = names only; 0 = none.

Chapter synthesis

An outcome that improves after a change has many possible causes. Hold several working hypotheses, and write each as a mechanism with a distinct prediction and data that can test it. Draw a causal diagram to decide what to adjust for, and avoid adjusting for mediators and colliders. Difference-in-differences removes common shocks and stable differences but not co-interventions; standardization removes only measured confounding; negative controls warn of what remains. When the historical data cannot separate the favoured explanation from its rivals, say so and choose a design that breaks the links — then list the new rivals that design introduces.

Continue studying — chapter 04 of 08

← Chapter 03 · Course chapter index · Chapter 05 →

Continue to the next authored lesson.