Chapter 10 — Research reading and critique
Lesson map
Lesson 10 published. Map, example, EX046.
flowchart TD
N0["Learning outcomes and connection"]
N1["10.1 Kinds of evidence about response effectiveness"]
N2["10.2 A reading protocol"]
N3["10.3 Typical biases"]
N4["10.4 Transfer and local validation"]
N5["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX046"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["10.1 Kinds of evidence about response effectiveness"]
N2["10.2 A reading protocol"]
N3["10.3 Typical biases"]
N4["10.4 Transfer and local validation"]
N5["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX046"]
[Learning outcomes and connection]
|
v
[10.1 Kinds of evidence about response effectiveness]
|
v
[10.2 A reading protocol]
|
v
[10.3 Typical biases]
|
v
[10.4 Transfer and local validation]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX046]
Learning outcomes and connection
You will be able to (1) read a published response-effectiveness study for design, population, outcome and limitations; (2) distinguish industry surveys, vendor incident datasets and academic studies; (3) judge whether a finding transfers to your organization; and (4) turn a published claim into a local validation question. Specialists are asked "what does the evidence say?" constantly; this chapter teaches the reading that the answer needs. Chapter 11 uses it to justify a program redesign.
10.1 Kinds of evidence about response effectiveness
- Industry surveys and cost studies — for example IBM's annual Cost of a Data Breach report (research conducted by the Ponemon Institute), which interviews organizations that experienced breaches and estimates costs, comparing groups by practices such as incident-response planning and testing.
- Incident-response provider datasets — for example Mandiant's annual M-Trends, which reports metrics such as median dwell time across incidents the firm investigated.
- Breach corpora — for example Verizon's Data Breach Investigations Report, built from contributed incident and breach data.
- Academic studies — controlled experiments, observational studies of organizations, qualitative studies of teams. Each has typical strengths and biases; none is "the evidence" alone.
10.2 A reading protocol
For any study, record: (1) question; (2) population and how it was sampled; (3) unit of analysis (incident, organization, analyst); (4) outcome and how measured; (5) comparison and design; (6) confounders addressed; (7) results with uncertainty; (8) limitations stated and unstated; (9) funding and interests; (10) applicability to your setting.
Applied to the two Calder readings, the protocol exposes their limits quickly. The cost report's population is breached organizations that agreed to be interviewed, with costs from an activity-based model; the provider report's population is incidents that one provider investigated, and its client mix changes each year. Neither is Calder's population of 400 mostly small, internally handled incidents a year.
10.3 Typical biases
- Selection: provider datasets include only incidents the provider was hired for — usually larger, more severe, or in organizations that buy such services. Survey participants are organizations willing to participate.
- Survivorship and detection: dwell time can only be measured for detected intrusions; an improvement in median dwell time can reflect better detection of short intrusions, more ransomware (which announces itself) or a changing mix.
- Confounding: organizations with tested IR plans also tend to have more security budget, maturity and staff; lower breach cost may reflect all of these.
- Measurement: costs estimated by models and interviews; categories change between editions.
- Interest: vendors publish data that may support their products; this does not make the data wrong, but readers should look for it.
10.4 Transfer and local validation
A finding transfers when the mechanism plausibly operates in your setting and your population resembles the study's. Rather than adopting a headline number ("IR testing saves $X"), formulate a local question your own data can address: "Did our tabletop-tested playbooks shorten our time from detection to containment for comparable incidents?" — with the methods of bachelor's Chapter 33 and master's-level design if needed.
Calder's local question follows this approach: for incidents of severity 2 or higher in 2025–2026, is time from detection to containment shorter for incident types with a tabletop-tested playbook, after accounting for incident type and division? Calder's own case records can answer it, which no external report can.
10.5 Worked example: two published sources, different populations
Sources (read the current editions yourself; figures change annually): IBM, Cost of a Data Breach Report; Mandiant (Google Cloud), M-Trends.
Claim type A (cost report): organizations with incident-response teams and regularly tested IR plans report lower average breach costs than those without. Reading: population = breached organizations that agreed to be interviewed, across countries and industries; unit = breach; outcome = estimated total cost via an activity-based model; comparison = cross-sectional groups by practice; confounders = partially addressed by reporting factors' associations with cost; limitations: self-selection, model-based costs, correlation not causation, practices co-occur.
Claim type B (provider report): global median dwell time has declined over the years. Reading: population = incidents investigated by the provider; unit = intrusion; outcome = time from earliest evidence of compromise to detection; comparison = across years; limitations: provider client mix changes, detection of intrusions that are never found is impossible, ransomware share affects the median, definitions of "earliest evidence" depend on available logs.
Transfer to Calder Group (synthetic): Calder's 400 incidents a year are mostly small and internally handled — unlike B's population. Calder is a large retailer and logistics firm; A's sample includes similar organizations but cost drivers (regulation, region) differ. Local validation question: "For Calder incidents of severity 2 or higher in 2025–2026, is time from detection to containment shorter for incident types with a tabletop-tested playbook than for those without, after accounting for incident type and division?"
10.6 Common errors and limits
- Quoting headline numbers without population and method.
- Treating year-over-year changes in provider datasets as global trends.
- Inferring causation from cross-sectional surveys.
- Dismissing vendor research entirely; it is often the only large-scale data — read it carefully instead.
The two reports read for Calder show where each error comes from. Quoting the cost report's lower average breach cost for tested plans as a saving would ignore self-selection and a cross-sectional comparison. Treating the provider's falling dwell time as a global trend would ignore its changing client mix. Yet both are worth reading: they suggest the local question Calder can test with its own data.
Exercises — chapter 10
Attempt each exercise before opening the worked results. Every case, organization, person, address and number in an exercise is synthetic unless a cited public source is named.
EX046 — Protocol on a real report. Steps: apply the ten-point protocol to the current edition of one of the two reports, citing page numbers. Rubric: 3 = all ten points with page references and unstated limitations identified; 2 = seven; 1 = summary; 0 = none.
EX047 — Dwell-time mechanisms. Steps: list four mechanisms other than "defenders got better" that could reduce median dwell time in a provider dataset. Rubric: 3 = ransomware share, client mix, earlier engagement, log retention changes, definition changes; 2 = three; 1 = one; 0 = none.
EX048 — Confounding diagram. Steps: draw a causal diagram for "tested IR plan → lower breach cost" with at least three confounders. Rubric: 3 = budget, maturity, industry/regulation, size, insurance; correct arrows; 2 = two confounders; 1 = no confounders; 0 = none.
EX049 — Local question. Steps: write a local validation question and design sketch for the claim "automation reduces breach lifecycle". Rubric: 3 = bounded question, outcome definition, comparison, confounders, data source, limitation; 2 = missing comparison; 1 = restated claim; 0 = none.
EX050 — Executive summary of evidence. Steps: write 120 words for executives answering "does IR testing pay off?" Rubric: 3 = honest synthesis: consistent association in surveys, causal evidence limited, plausible mechanisms, local measurement plan; 2 = overclaims; 1 = headline number only; 0 = none.
Chapter synthesis
Read every study for population, unit, outcome, comparison and interests; recognize selection, detection and confounding biases typical of industry data; transfer findings only through plausible mechanisms; and convert published claims into local validation questions.
Continue studying — chapter 10 of 12
← Chapter 09 · Course chapter index · Chapter 11 →
Continue to the next authored lesson.