Chapter 10 — Research reading and critique

Lesson map

Lesson 10 published. Map, example, EX046.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["10.1 Three kinds of source"]
  N2["10.2 Reading a study: design first"]
  N3["10.3 Reading an advisory"]
  N4["10.4 Reading a vendor claim"]
  N5["10.5 Confounding and Simpson's paradox"]
  N6["10.6 Transfer"]
  N7["10.7 The evidence brief"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX046"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["10.1 Three kinds of source"]
  N2["10.2 Reading a study: design first"]
  N3["10.3 Reading an advisory"]
  N4["10.4 Reading a vendor claim"]
  N5["10.5 Confounding and Simpson's paradox"]
  N6["10.6 Transfer"]
  N7["10.7 The evidence brief"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX046"]

[Learning outcomes and connection]
    |
    v
[10.1 Three kinds of source]
    |
    v
[10.2 Reading a study: design first]
    |
    v
[10.3 Reading an advisory]
    |
    v
[10.4 Reading a vendor claim]
    |
    v
[10.5 Confounding and Simpson's paradox]
    |
    v
[10.6 Transfer]
    |
    v
[10.7 The evidence brief]
    |
    v
[Worked example] --> [Exercise EX046]

Learning outcomes and connection

After this chapter you should be able to:

1. Read a published study, an official advisory and a vendor claim for what it can contribute to a practical decision.
2. Identify the design of a study and the strongest claim that design supports.
3. Check denominators, comparisons and outcome definitions, and detect confounding such as Simpson's paradox.
4. Judge whether a finding transfers to your organization and sector.
5. Write a short evidence brief that states what a source shows, what it does not, and what you will do with it.

A specialist does not need to produce research, but must use it: to choose controls, to set probabilities (Chapters 1 and 4), to argue with suppliers and to brief boards. This chapter teaches reading for decisions. Chapter 11 then applies the same critical eye to evaluating your own project.

10.1 Three kinds of source

Lesson table; scroll horizontally to read all columns →

None is automatically trustworthy or worthless. Each answers some questions well and others badly. Your job is to match the source to the question.

10.2 Reading a study: design first

Before reading results, identify the design, because it limits the claim. A before-and-after comparison at one site can show that something changed but struggles to show why. A comparison with an unaffected control group, or an interrupted time series, supports stronger causal language. A survey of victims cannot estimate how often attacks happen in the whole population. Then check: the unit (hospital, patient, day), the outcome and how it was measured, the comparison (what would have happened otherwise), the time window, and the setting.

Consider two real health-sector studies. Ghafur and colleagues analysed national hospital activity data around the 2017 WannaCry attack and found no significant change in total activity across all English trusts, but about 6% fewer admissions per day at infected hospitals, an estimated £5.9 million of lost activity, and no detected difference in mortality — while noting that mortality is a crude measure of patient harm (Ghafur et al., 2019). Dameff and colleagues compared emergency-department volumes at two hospitals next to a health system under a month-long ransomware attack in 2021, before, during and after the attack, and reported significant increases in census, ambulance arrivals and other measures during the attack (Dameff et al., 2023). Both are valuable; both are bounded by design. The first uses national data with comparison across trusts; the second is a before-and-after study at one institution, so seasonal or local changes cannot be fully excluded. Neither tells Merrow Health how likely an attack is; both inform how severe and how regional the consequences can be.

Population-level datasets answer different questions. Neprash and colleagues used a database of publicly reported ransomware attacks on U.S. health care delivery organizations and found that annual attacks more than doubled from 2016 to 2021 and that almost half disrupted care delivery (Neprash et al., 2022). Because the data are publicly reported attacks, trends combine changes in attacks with changes in reporting — a limitation the authors' design makes explicit and a reader must carry forward.

10.3 Reading an advisory

Advisories such as CISA's joint advisory on state-sponsored pre-positioning in U.S. critical infrastructure (AA24-038A, 2024) give techniques, indicators and mitigations from investigated cases. Read them for: what was observed (techniques, targeted sectors, entry vectors); what is recommended (mitigations, often prioritized); and what is inferred (intent, scale). Translate the observed techniques into questions about your own environment — "would we see this?" — rather than into a probability. An advisory is evidence that a threat exists and how it operates, not an estimate of how often it succeeds.

10.4 Reading a vendor claim

Vendor claims deserve the same reading discipline. Ask: what is the outcome (reported incidents? detected incidents? losses?); what is the denominator (customers, sites, operator-years?); who is in each group and why (self-selection, size, sector); and what would a detection product do to the outcome (more monitoring finds more incidents). Research on machine learning in security lists pitfalls — sampling bias, data snooping, inappropriate baselines, base-rate fallacy, unrealistic threat models — that apply to many product evaluations (Arp et al., 2022).

10.5 Confounding and Simpson's paradox

When groups differ in a factor that also affects the outcome, a pooled comparison can point in the opposite direction from every within-group comparison. This is Simpson's paradox (Simpson, 1951). In security data, size is a frequent confounder: large operators have more systems, more reporting obligations and more incidents, and they buy products in different proportions from small ones. The remedy is to stratify (compare within size bands) or standardize (re-weight groups to the same mix) and to ask whether the outcome measures harm or detection.

10.6 Transfer

A finding transfers to your setting only if the mechanism that produced it operates there. The formal literature on transportability asks which population differences change the effect (Pearl and Bareinboim, 2014); in practice, list the conditions the finding depended on — size, sector, regulatory reporting, technology, staffing — and check each for your organization. A mortality finding from a national health system with network-wide diversion arrangements may not transfer to a rural region with one hospital.

For the vendor brochure in this chapter, transfer is a further question after confounding: even a well-designed comparison among other operators would apply to Merrow Bay only if their size, reporting practice and plant resembled the forum's members. Write the conditions down and mark each as checked, unknown or different.

10.7 The evidence brief

Summarize each important source for decision-makers in a short evidence brief: the question it helps with; design and setting; main finding with its size and uncertainty; strongest claim supported; key limitations and their likely direction; transfer to our setting; and what we will do with it. Keep briefs in a library so later decisions can cite them (Chapter 1's decision records).

A brief is short by design, usually one page, so that a director can read it before a meeting. The strongest claim supported should be written in the words you would be willing to defend in front of the authors; if the source's own conclusion is stronger, say so and explain the difference. Date every brief and note when it should be reviewed, since advisories and vendor data age quickly.

10.8 Worked example: a monitoring vendor's brochure (synthetic)

A vendor of OT monitoring tells the resilience forum: "Customers using our monitoring had 40% fewer reported incidents." Asked for data, the vendor supplies counts by operator size. Lab 10 re-reads them.

The specialist's first move is to ask what lies behind the number before arguing about it: what counts as an incident, over how many operator-years, and how the customers and non-customers differ. The vendor's willingness to supply counts by size is itself useful; a vendor that refuses to share any denominator has given the forum a reason to treat the claim as marketing. The forum agrees to review the data together rather than respond to the brochure.

10.9 Practical activity: aggregation, denominators and detection

# Lab 10 - re-reading a vendor claim: aggregation, denominators and survivorship (offline, stdlib only)
# Synthetic: a vendor brochure says "customers using our OT monitoring had 40% fewer reported incidents".
# Below are the (invented) counts behind such a claim, split by operator size.
DATA = {  # stratum: {group: (operators, operator-years observed, incidents reported)}
    "small (<50 staff)": {"monitoring": (240, 720, 54), "none": (100, 300, 15)},
    "large (>=50 staff)": {"monitoring": (20, 60, 18),   "none": (100, 300, 75)},
}
def rate(g):
    oy = sum(DATA[s][g][1] for s in DATA); inc = sum(DATA[s][g][2] for s in DATA)
    return inc / oy, inc, oy
for g in ("monitoring", "none"):
    r, inc, oy = rate(g); print(f"pooled {g:10s} {inc:3d} incidents / {oy:4d} operator-years = {r:.3f}")
rm, rn = rate("monitoring")[0], rate("none")[0]
print(f"pooled relative difference: {(rm - rn) / rn:+.0%}")
for s, grp in DATA.items():
    a = grp["monitoring"][2] / grp["monitoring"][1]; b = grp["none"][2] / grp["none"][1]
    print(f"{s:20s} monitoring {a:.3f}  none {b:.3f}  ratio {a/b:.2f}")
# Standardize both groups to the same size mix (the 'none' group's operator-years) - direct standardization
w = {s: DATA[s]["none"][1] for s in DATA}; W = sum(w.values())
std = {g: sum(w[s] / W * DATA[s][g][2] / DATA[s][g][1] for s in DATA) for g in ("monitoring", "none")}
print(f"size-standardized: monitoring {std['monitoring']:.3f}  none {std['none']:.3f}  ratio {std['monitoring']/std['none']:.2f}")

Expected output:

pooled monitoring  72 incidents /  780 operator-years = 0.092
pooled none        90 incidents /  600 operator-years = 0.150
pooled relative difference: -38%
small (<50 staff)    monitoring 0.075  none 0.050  ratio 1.50
large (>=50 staff)   monitoring 0.300  none 0.250  ratio 1.20
size-standardized: monitoring 0.188  none 0.150  ratio 1.25

Interpretation. Pooled, monitored operators report 0.092 incidents per operator-year against 0.150 for unmonitored operators, a 38% difference — the brochure's claim. But within each size band, monitored operators report more incidents: 1.5 times as many among small operators, 1.2 times among large ones. The pooled comparison reverses because most monitored operators are small, and small operators report fewer incidents whatever they use. Standardized to the same size mix, monitored operators report 1.25 times as many incidents. And "reported incidents" is a detection outcome: a monitoring product that works should increase reported incidents by finding what was missed. Neither the pooled nor the stratified figures say anything about harm prevented. The right response to the vendor is not "your product is bad" but "your claim does not follow from these data; show us outcomes in service units, matched by size and sector, over time".

10.10 Common errors and limits

• Reading results before identifying the design.
• Treating an advisory's case list as a frequency estimate.
• Comparing groups that differ in size or sector without stratifying.
• Treating reported incidents as a measure of harm or of security.
• Accepting a finding without checking whether its mechanism operates in your setting.
• Dismissing a source entirely because it has one weakness.
• Citing a source for more than it showed.

The limit of critical reading is time. Prioritize sources that could change a decision (Chapter 1's value of information), read those carefully, and accept lighter reading for the rest.

Exercises — chapter 10

EX046 — Re-read the brochure. Objective: reproduce and explain a reversal. Inputs: Lab 10. Constraints: 250 words for the explanation. Steps: (1) run the lab; (2) explain the reversal to the forum in plain language; (3) propose two outcome measures that would better show the product's value. Expected evidence: output and explanation. Rubric: 3 = reversal explained via size mix and detection effect, standardized ratio quoted, outcome measures tied to harm (for example hours of service interruption, time to detect injected tests); 2 = reversal explained, measures weak; 1 = repeats the brochure; 0 = none.

EX047 — Evidence brief: neighbouring hospitals. Objective: brief a study for Merrow Health. Inputs: Dameff et al. (2023) abstract. Constraints: use §10.7 structure; quote numbers only from the source. Steps: write the brief. Expected evidence: one-page brief. Rubric: 3 = design identified as before-and-after at one institution, findings stated with their measures, limitations (single site, seasonality, no control region) with direction, transfer to Merrow Bay's three-hospital system discussed, action (for example regional diversion planning) stated; 2 = limitations undirected; 1 = over-claims causality; 0 = none.

EX048 — Evidence brief: WannaCry. Objective: brief a national study. Inputs: Ghafur et al. (2019) and the NAO investigation. Constraints: distinguish the two sources' purposes. Steps: write a brief answering "what does WannaCry tell Merrow Health about the consequences of a large-scale ransomware event?" Expected evidence: brief. Rubric: 3 = study's findings (about 6% fewer admissions per day at infected hospitals; no detected mortality difference with the authors' caution) and NAO's audit purpose distinguished, transfer issues stated, action stated; 2 = sources merged; 1 = "no deaths so low risk"; 0 = none.

EX049 — From advisory to questions. Objective: use an advisory well. Inputs: CISA AA24-038A. Constraints: do not estimate probability from it. Steps: (1) extract five observed techniques or behaviours; (2) for each, write the question "would we see this at Merrow Water?" and the evidence source that would answer it; (3) pick two recommended mitigations and map them to Chapter 2's practices. Expected evidence: table. Rubric: 3 = observed behaviours accurately summarized (for example valid-account use and living-off-the-land techniques), detection questions tied to specific logs, mitigations mapped; 2 = mixes observation and inference; 1 = summary only; 0 = none.

EX050 — Pitfall check. Objective: apply a research checklist to a product evaluation. Inputs: a vendor states its machine-learning anomaly detector "achieved 99% accuracy on a public water-treatment testbed dataset". Constraints: use pitfalls from Arp et al. (2022). Steps: list five questions you would ask and why each matters. Expected evidence: list. Rubric: 3 = base-rate and accuracy-versus-precision issue, train/test leakage (data snooping), testbed-to-plant transfer, baseline comparison, threat model realism; 2 = three; 1 = generic scepticism; 0 = none.

Chapter synthesis

Read for decisions. Match the source to the question: studies for effect sizes and mechanisms, advisories for techniques and mitigations, vendor reports for hypotheses to verify. Identify design before results and limit claims to what the design supports. Check outcome, denominator, comparison and confounders; stratify or standardize when groups differ, and remember that reported incidents measure detection as much as harm. Ask whether the mechanism transfers to your setting. Record what you learn in evidence briefs that decision records can cite.