◇ OpenCyber Academy
Master's · Chapter 1 · English

Chapter 1 — Research question and literature method

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 1 — Research question and literature method

Lesson map

Lesson 01 published. Map, example, EX001.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["1.1 From topic to problem to question"]
  N2["1.2 The estimand: say exactly what number you will p"]
  N3["1.3 What a literature review is for"]
  N4["1.4 Searching across disciplines"]
  N5["1.5 From records to included studies"]
  N6["1.6 Synthesis by concept, not by author"]
  N7["1.7 Reading a study for design, not only for finding"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX001"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["1.1 From topic to problem to question"]
  N2["1.2 The estimand: say exactly what number you will p"]
  N3["1.3 What a literature review is for"]
  N4["1.4 Searching across disciplines"]
  N5["1.5 From records to included studies"]
  N6["1.6 Synthesis by concept, not by author"]
  N7["1.7 Reading a study for design, not only for finding"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX001"]

[Learning outcomes and connection]
    |
    v
[1.1 From topic to problem to question]
    |
    v
[1.2 The estimand: say exactly what number you will p]
    |
    v
[1.3 What a literature review is for]
    |
    v
[1.4 Searching across disciplines]
    |
    v
[1.5 From records to included studies]
    |
    v
[1.6 Synthesis by concept, not by author]
    |
    v
[1.7 Reading a study for design, not only for finding]
    |
    v
[Worked example] --> [Exercise EX001]

Learning outcomes and connection

By the end of this chapter the learner will be able to:

  1. Move from a topic ("ransomware in hospitals") through a problem to a single answerable research question, and test it against the FINER criteria and an infrastructure version of PICOT.
  2. Classify a question as descriptive, associational, causal, explanatory or evaluative, and state the kind of design and claim each permits.
  3. Define the quantity the study will estimate — its estimand — in words precise enough that two analysts would compute the same thing.
  4. Choose a review type (narrative, scoping, systematic) appropriate for a master's dissertation and justify it.
  5. Build, run and document a cross-disciplinary search that includes grey literature, test it against known items, and stop it by a predeclared rule.
  6. Deduplicate records, screen them with two reviewers, measure agreement with Cohen's kappa, repair eligibility criteria where disagreement concentrates, and report a flow diagram.
  7. Organize what was found in a concept matrix rather than an author-by-author summary.

The chapter sets the question and the evidence map that every later chapter uses. Chapter 2 widens the map to the other sectors; Chapter 3 chooses a design for the question fixed here; Chapter 4 lists rival explanations the literature has already raised; Chapter 7 writes the literature review chapter from the concept matrix built here.

1.1 From topic to problem to question

1.1.1 Three different things

Most dissertations begin with a topic, which is an area of interest: "cyberattacks on hospitals". A topic cannot be answered. A problem is a gap between what is known and what someone needs to know in order to act: "regional emergency planners do not know how much a cyber-induced diversion at one hospital burdens its neighbours, so mutual-aid agreements are written without numbers." A research question is the specific question whose answer would reduce that gap: "During a ransomware-induced ambulance diversion, how much do daily ambulance arrivals change at hospitals within 40 km, relative to what would have been expected without the diversion?"

The movement from topic to question is a narrowing, and every narrowing is a decision you must be able to defend. Narrowing to ambulance arrivals rather than "patient harm" trades importance for measurability. Narrowing to one region trades generality for depth and data access. Narrowing to ransomware-induced diversion rather than "cyber incidents" makes the exposure observable: a diversion has a start and an end that an emergency medical services (EMS) agency records, whereas "a cyber incident" may be unknown even to the hospital for weeks.

1.1.2 FINER and an infrastructure PICOT

Clinical researchers test questions against the FINER criteria — feasible, interesting, novel, ethical and relevant — and structure them with PICOT: population, intervention, comparison, outcome and time (Farrugia et al., 2010, summarizing Hulley and colleagues' Designing Clinical Research). Both transfer to infrastructure research with one change: most infrastructure studies observe an exposure that the researcher did not assign, so the "I" becomes "E".

Element Meaning in infrastructure research Spillover Study (first version)
Population / units The service-delivering units studied: hospitals, emergency departments, feeders, water systems, payment processors. State the unit of analysis and the unit of observation (they can differ: the unit might be a hospital-day). Emergency departments in Tamsin Valley other than Kestrel's; unit of observation: the ED-day.
Exposure The event or condition whose consequence is studied, defined so that its start and end are observable. Ambulance diversion at Kestrel caused by a ransomware attack (days 0–20).
Comparison What the exposed experience is compared with: the same units before, unexposed units at the same time, or both. The same EDs in the 180 days before; Wexmoor EDs over the same days.
Outcome A consequence measured in service units. Name the primary outcome and at most a few secondary ones. Primary: daily ambulance arrivals. Secondary: median minutes from arrival to first clinician.
Time Window, follow-up and resolution (hourly, daily, monthly). Daily, from day −180 to day +89.

FINER is a screen, not a score. Feasible means the data exist, you can obtain them lawfully within the course's time, and the analysis is within your competence. Interesting means someone who must make a decision cares about the answer. Novel at master's level includes "measured for the first time in this setting" and "replicated with a stronger design". Ethical means a review board could approve it (Chapter 3). Relevant means the answer would change a plan, a policy or a later study.

1.1.3 Question types and the claims they permit

Type Form Example What the design must provide Strongest claim
Descriptive How much, how often, what? How many hours of ambulance diversion did Tamsin Valley EDs record last year, by cause? A defined population and complete records. A measured quantity in that population.
Associational Is X associated with Y? Are diversion hours higher at EDs closer to a diverting hospital? Variation in X and Y; adjustment for obvious confounders. An association, with its size and uncertainty.
Causal How much did X change Y? How much did Kestrel's diversion increase arrivals at Alder Point? A credible counterfactual: what would have happened without X (Chapters 3–4). An effect estimate under stated assumptions.
Explanatory Why or how did X change Y? Through which decisions did dispatchers redistribute patients? Process evidence: interviews, documents, logs. A mechanism supported by converging accounts.
Evaluative Did a measure work? Did the coalition's diversion protocol shorten waiting times? A causal design plus an operational definition of "work". An effect of the measure, under assumptions.

The Spillover Study's primary question is causal, its secondary question explanatory. That combination — a quantitative estimate plus an account of mechanism — is typical of a strong master's dissertation and is the reason this book teaches both strands.

1.2 The estimand: say exactly what number you will produce

A question in words still leaves room for many different numbers. "How much do arrivals rise?" could mean the rise on the first day, the average rise over the diversion, the rise relative to last year, the rise relative to a comparison region, or the total extra patients. An estimand is the precise quantity a study aims to estimate, defined before data are seen. A useful template has five parts:

  1. Units: which units and which days (Alder Point ED, days 0–20).
  2. Outcome: what is counted and how (ambulance arrivals per calendar day, as recorded by the ED's arrival log).
  3. Contrast: compared with what (the arrivals that would have occurred in the absence of the diversion).
  4. Summary: the scale (ratio of the observed to the counterfactual mean, reported as a percentage change).
  5. Population of inference: whom the answer is about (this ED during this event — not "hospitals in general").

The Spillover Study's primary estimand is therefore: for each neighbouring ED, the percentage change in mean daily ambulance arrivals during days 0–20, relative to the arrivals expected from that ED's own pre-event trend and the concurrent change at unaffected Wexmoor EDs. The counterfactual "would have occurred" is never observed; Chapters 3 and 4 decide how to approximate it. Writing the estimand first prevents a common failure: choosing the comparison after seeing which one gives the most impressive number.

1.2.1 Claim scope

The doctoral book in this area describes a five-rung ladder from observation to research claim. At master's level the essential discipline is simpler: write the claim you hope to make and the scope it will have before you start, then check at the end whether the evidence reached it. The Spillover Study hopes to claim: "In one synthetic regional event, ambulance arrivals at the nearest neighbouring ED rose during the diversion by an amount we estimate with stated uncertainty, and the rise was smaller at more distant EDs." It does not hope to claim that ransomware harms patients, that all regions behave this way, or that diversion policies should change — although the discussion may suggest which studies could support those larger claims.

1.3 What a literature review is for

At master's level the review has four jobs: to show that the question is open or worth re-asking; to find methods and measures others have used, so you do not reinvent them; to find rival explanations already raised (Chapter 4); and to place the eventual result in context. It is not an annotated bibliography and it is not a demonstration of how much you read.

Review type Purpose Method requirements Fits a master's project when
Narrative (traditional) Expert synthesis and argument Few formal rules; risk of selective citing The review supports a primary empirical study and the field is small
Scoping Map the extent, range and nature of evidence; clarify concepts Framework of Arksey and O'Malley (2005); reporting by PRISMA-ScR (Tricco et al., 2018) The literature is scattered across disciplines and the question is still being shaped
Systematic Answer a focused question with exhaustive search and appraisal Protocol, dual screening, risk of bias, PRISMA 2020 reporting The review is the dissertation, and there is enough literature to synthesize
Integrative / semi-systematic Combine diverse study types into a conceptual synthesis Documented search with flexible synthesis (Snyder, 2019) The field mixes empirical, technical and policy work

For the Spillover Study the review is a structured scoping review supporting an empirical study: documented search, explicit eligibility, two reviewers on a sample, a flow diagram and a concept matrix — but no formal risk-of-bias scoring or meta-analysis, because the handful of relevant studies use incomparable designs. State this choice and its reason in the methods chapter; a reader should never have to guess whether a review was meant to be exhaustive.

1.4 Searching across disciplines

1.4.1 Why infrastructure literature is scattered

Evidence on the Spillover question is spread across emergency medicine, health-services research, health economics, disaster medicine, information security, public administration and government audit. The same idea carries different names: "ambulance diversion", "ED crowding", "regional surge", "patient redistribution", "blast radius". Security databases rarely index hospital operations; medical databases rarely index security conference papers; and some of the most informative evidence — audit reports, regulator statements, after-action reports, healthcare-coalition exercise reports — is grey literature that no bibliographic database indexes.

1.4.2 Concept blocks

Build the search from concept blocks combined with AND; inside each block, synonyms combined with OR:

Including "downtime" and "IT outage" in Block A is a deliberate choice: non-malicious outages produce similar operational effects and may supply methods and rivals (Chapter 4). Record the choice; it expands screening work and must be handled by eligibility criteria.

1.4.3 Sources and the known-item test

Search at least one biomedical index, one general scholarly index and one computing index, plus a grey-literature plan: government audit offices, health ministries and agencies, sector coordinating bodies, hospital associations and coalition reports. Before running the full search, assemble a known-item set — studies you already know are relevant — and check that your search finds them. For the Spillover question a reasonable known-item set is Dameff et al. (2023), Ghafur et al. (2019) on WannaCry in the English NHS and Neprash et al. (2022) on ransomware trends in U.S. health care delivery organizations. A search that misses a known item is too narrow; revise it and log the revision.

1.4.4 Citation chaining and stopping

Database searches miss papers that use unexpected vocabulary. Backward chaining reads the reference lists of included studies; forward chaining finds later papers that cite them. Wohlin (2014) gives guidelines for this "snowballing" and shows it can complement or substitute for database search in some fields. Decide in advance when to stop: for example, after two consecutive chaining rounds that add no included study. Record every search in a search log: date, source, exact string, filters, number of hits, and who ran it.

1.5 From records to included studies

1.5.1 Deduplication

The same paper appears in several databases, with different capitalization, punctuation, hyphenation and sometimes without a DOI. Deduplicate before screening, so that two reviewers do not screen the same study twice and the flow diagram is honest. A sound two-pass procedure is: (1) match exact DOIs after lower-casing; (2) match normalized titles plus year among the remainder — lower-case, strip punctuation, collapse spaces and normalize known variants (for example "turn-around" and "turnaround"). Inspect every automatic match by eye on a small project; automatic rules can merge different papers with generic titles ("Editorial", "Correction").

1.5.2 Two-stage screening and agreement

Screen titles and abstracts against eligibility criteria, then read full texts of everything that passes. At master's level, two reviewers often cannot screen everything; a defensible compromise is that a second reviewer (a peer or the supervisor) independently screens a sample, agreement is measured, criteria are repaired where disagreement concentrates, and the first reviewer then screens the rest with the repaired criteria.

Agreement is measured with Cohen's kappa (Cohen, 1960), which corrects observed agreement pₒ for the agreement pₑ expected by chance from each reviewer's include rate: κ = (pₒ − pₑ) / (1 − pₑ). Two reviewers who each include half the records would agree half the time by chance alone, so 80% raw agreement may be unimpressive. Interpretation bands vary, and McHugh (2012) argues that the widely quoted bands are too lenient for health research, where low agreement means unreliable data. Treat kappa as a diagnostic: where reviewers disagree tells you which criterion is unclear.

1.5.3 The flow diagram

Report records identified per source, duplicates removed, records screened, records excluded, full texts assessed, full texts excluded with reasons, and studies included. PRISMA 2020 provides a standard flow diagram (Page et al., 2021). Even a non-systematic review benefits from it, because it lets an examiner see the denominator behind "the literature shows".

1.6 Synthesis by concept, not by author

Webster and Watson (2002) recommended that reviews be concept-centric rather than author-centric: instead of a paragraph per paper ("Smith found…; Jones found…"), organize the review around the concepts the question needs and show which studies inform each. A concept matrix has studies as rows and concepts as columns. For the Spillover Study the columns are: unit of analysis; exposure definition; comparison; outcome; design; what the study can support; rivals it raises.

Study Unit Exposure Comparison Outcomes relevant to spillover Design (as published) What it can support
Dameff et al., 2023 Two EDs adjacent to an attacked organization Month-long ransomware attack on four adjacent hospitals (May 2021) The same EDs in the four weeks before and after Census, ambulance arrivals, waiting times, left without being seen, length of stay, county EMS diversion, stroke metrics Before-and-after cohort An association between the attack period and burden at neighbouring EDs in one region
Ghafur et al., 2019 English NHS trusts WannaCry (May 2017) Baseline weeks; infected versus non-infected trusts Admissions, A&E attendances, outpatient cancellations, A&E deaths Retrospective analysis of Hospital Episode Statistics Activity reductions at infected trusts; no detectable change in A&E deaths (a crude harm measure, as the authors note)
Neprash et al., 2022 U.S. health care delivery organizations Ransomware attacks in a database built from public reports, 2016–2021 Time trends Frequency, PHI exposure, operational disruption including ambulance diversion Cohort of reported attacks Trends and characteristics of reported attacks

The matrix makes the gap visible: one before-and-after study of neighbouring EDs, no study with an unaffected comparison region, and no study of whether the burden decreases with distance. That is the Spillover Study's justification.

1.7 Reading a study for design, not only for findings

When you extract a study, record not only what it found but what it could have found. Ask: What is the unit? Is the exposure defined by its start and end? What is the comparison, and what else changed between the compared periods? Is the outcome a consequence (patients delayed) or an activity (visits counted)? What population could the result generalize to? A before-and-after comparison within the same units, for example, cannot separate the exposure from anything else that changed at the same time — season, policy, staffing, a concurrent epidemic. Saying this is not criticism of the authors; it is the evidence your own design must improve upon.

1.8 Worked example: from topic to the Spillover question (synthetic)

The candidate's first proposal paragraph read: "This dissertation will investigate the impact of cyberattacks on hospitals and show that they endanger patients across the region." The supervisor's review:

Problem Why it fails Revision
"Investigate the impact of cyberattacks on hospitals" A topic, not a question; "impact" is undefined. Name the outcome: daily ambulance arrivals and waiting time at neighbouring EDs.
"Show that they endanger patients" Presupposes the answer; a design to "show" cannot fail. Ask "how much" and allow zero as an answer.
"Across the region" No unit or boundary. EDs within 40 km of the attacked system; distance as a moderator.
"Cyberattacks" Exposure not observable or timed. Ambulance diversion caused by a ransomware attack, with EMS-recorded start and end.
No comparison Before-and-after alone cannot separate the attack from the season. Add an unaffected comparison region.
No mechanism Even a credible number would not tell planners what to change. Secondary explanatory question with interviews of ED, EMS and coalition staff.
"Patients are endangered" Patient outcomes need clinical data and approvals beyond a master's timeline. Keep patient outcomes out of scope; name them as future work.

Final questions:

1.9 Practical activity: deduplicate, screen and measure agreement

The lab below holds a synthetic export of 20 records from three databases and a grey-literature search. It deduplicates them in two passes, takes two reviewers' title-and-abstract decisions on the remaining records, computes kappa and lists disagreements. Run it with Python 3; no packages are needed.

# Lab 1 - deduplicate a synthetic search export, measure screening agreement, count the flow (offline)
# Each record: (source, title, year, doi_or_empty). Decisions: reviewer A and B, I = include, X = exclude.
import re

RECORDS = [
    ("medline", "Ransomware and emergency department crowding at neighbouring hospitals", 2023, "10.9999/syn.001"),
    ("scopus",  "Ransomware and Emergency Department Crowding at Neighbouring Hospitals.", 2023, "10.9999/SYN.001"),
    ("ieee",    "Anomaly detection for water distribution SCADA", 2019, "10.9999/syn.002"),
    ("medline", "Hospital IT downtime and laboratory turnaround times", 2021, ""),
    ("scopus",  "Hospital IT downtime and laboratory turn-around times", 2021, "10.9999/syn.003"),
    ("grey",    "After-action report: regional EMS diversion during a hospital network outage", 2022, ""),
    ("scopus",  "Ambulance diversion policies and patient outcomes: a review", 2018, "10.9999/syn.004"),
    ("medline", "Ambulance diversion policies and patient outcomes - a review", 2018, ""),
    ("ieee",    "Cyber-physical attack graphs for substations", 2020, "10.9999/syn.005"),
    ("grey",    "State health department memo on downtime procedures", 2024, ""),
    ("scopus",  "Interrupted time series of ED arrivals after hospital closure", 2017, "10.9999/syn.006"),
    ("medline", "Mortality after hospital cyberattacks in Medicare beneficiaries", 2024, "10.9999/syn.007"),
    ("scopus",  "Mortality after hospital cyberattacks in Medicare beneficiaries", 2024, "10.9999/syn.007"),
    ("ieee",    "A survey of ransomware families 2015-2022", 2022, "10.9999/syn.008"),
    ("grey",    "Regional healthcare coalition cyber-incident exercise report", 2023, ""),
    ("medline", "Stroke care times during regional diversion", 2020, "10.9999/syn.009"),
    ("scopus",  "Electronic health record downtime: a scoping review", 2022, "10.9999/syn.010"),
    ("medline", "Electronic Health Record Downtime: A Scoping Review", 2022, "10.9999/syn.010"),
    ("ieee",    "Blockchain for hospital supply chains", 2021, "10.9999/syn.011"),
    ("grey",    "Vendor white paper: ransomware readiness in hospitals", 2023, ""),
]

def norm_title(t):
    t = t.lower().replace("turn-around", "turnaround")
    t = re.sub(r"[^a-z0-9 ]+", " ", t)
    return " ".join(t.split())

def key(rec):
    _, title, year, doi = rec
    return ("doi", doi.lower()) if doi else ("ty", norm_title(title), year)

seen, unique, dup = {}, [], 0
for rec in RECORDS:                       # pass 1: DOI or title+year
    k = key(rec)
    if k in seen:
        dup += 1
        continue
    seen[k] = rec
    unique.append(rec)
by_title = {}
final = []
for rec in unique:                        # pass 2: same normalized title+year, one record lacks a DOI
    tk = (norm_title(rec[1]), rec[2])
    if tk in by_title:
        dup += 1
        continue
    by_title[tk] = rec
    final.append(rec)

# Title/abstract screening decisions by two reviewers, in the order of `final`
A = "I X I I I X X I I X I I I X X".split()
B = "I X I I I X I I I X X I I X I".split()

def kappa(a, b):
    n = len(a)
    po = sum(x == y for x, y in zip(a, b)) / n
    cats = set(a) | set(b)
    pe = sum((a.count(c) / n) * (b.count(c) / n) for c in cats)
    return po, pe, (po - pe) / (1 - pe)

po, pe, k = kappa(A, B)
both = sum(x == y == "I" for x, y in zip(A, B))
disagree = [final[i][1] for i in range(len(final)) if A[i] != B[i]]
print(f"records identified: {len(RECORDS)}  duplicates removed: {dup}  screened: {len(final)}")
print(f"agreement po={po:.3f} pe={pe:.3f} kappa={k:.3f}")
print(f"both include: {both}  disagreements: {len(disagree)}")
for t in disagree:
    print("  resolve:", t)

Expected output:

records identified: 20  duplicates removed: 5  screened: 15
agreement po=0.800 pe=0.533 kappa=0.571
both include: 8  disagreements: 3
  resolve: State health department memo on downtime procedures
  resolve: Regional healthcare coalition cyber-incident exercise report
  resolve: Vendor white paper: ransomware readiness in hospitals

Five duplicates were removed: three by DOI (one with a differently capitalized DOI) and two by normalized title and year, including one pair that differed only by "turn-around" versus "turnaround" and one that differed only by a dash versus a colon. Raw agreement on the 15 screened records is 80%, but chance agreement is 53%, so κ = 0.571 — moderate at best. The diagnostic value is in the list: all three disagreements are grey literature (a health-department memo, a coalition exercise report and a vendor white paper). The criteria said nothing about grey literature. Repair them before screening continues: for example, include government, regulator, coalition and after-action reports that describe a specific event's operational effects; exclude vendor marketing and documents that describe procedures without reporting effects. Then re-screen the sample and recompute kappa.

1.10 Common errors and limits

The limit of a master's review is time: it will not be exhaustive, and it should not pretend to be. Its credibility comes from a documented, repeatable search and an honest statement of what was not searched.

Exercises — chapter 1

EX001 — From topic to question. Objective: narrow a topic into an answerable question. Inputs: the topic "cyber incidents at drinking-water utilities". Constraints: one primary question; an observable, timed exposure; an outcome in service units. Steps: (1) state a problem a decision-maker faces; (2) write a question in infrastructure PICOT form; (3) test it against each FINER criterion in one sentence; (4) state the claim you hope to make and its scope. Expected evidence: a half-page memo. Rubric: 3 = question answerable with plausible data, exposure and outcome observable, FINER honestly applied (including what is not feasible), claim scoped to the design; 2 = one element vague; 1 = still a topic; 0 = none.

EX002 — Question types. Objective: match questions to designs and claims. Inputs: (a) "How many hours of ambulance diversion were recorded in the county in 2024, by cause?" (b) "Do EDs that adopted downtime drills have shorter waiting times during IT outages?" (c) "How much did the diversion increase arrivals at Alder Point?" (d) "Why did dispatchers send stroke patients past the nearest open hospital?" (e) "Did the coalition's new diversion protocol reduce time on diversion?" Constraints: use the table in §1.1.3. Steps: classify each; name the minimum design; name the strongest claim. Expected evidence: table. Rubric: 3 = all classified correctly with designs that would actually support the claims, and (b) recognized as associational unless drill adoption can be treated as an exposure with a credible counterfactual; 2 = one error; 1 = designs generic; 0 = none.

EX003 — Search strategy and known-item test. Objective: build a documented, cross-disciplinary search. Inputs: RQ1–RQ3 in §1.8. Constraints: three concept blocks; at least three bibliographic sources from different disciplines and a grey-literature plan with at least four source types. Steps: (1) write the blocks; (2) list sources and justify each; (3) define a known-item set of three studies and describe how you would check that each is retrieved; (4) define a stopping rule for citation chaining; (5) design a search-log template. Expected evidence: one-page protocol and log template. Rubric: 3 = blocks include cross-disciplinary synonyms, grey literature specified by source type, known-item test and stopping rule concrete; 2 = one element missing; 1 = single-discipline search; 0 = none.

EX004 — Deduplication and agreement. Objective: test whether title-based deduplication catches a near-duplicate grey-literature record, and see how a single reviewer disagreement moves Cohen's kappa. Inputs: Lab 1 code. Constraints: offline. Steps: (1) run the lab and record the output; (2) add a record ("grey", "Ransomware & emergency department crowding at neighbouring hospitals", 2023, "") at the end of RECORDS, append I to both reviewers' decision strings (the record reaches screening only if deduplication misses it), rerun, and report whether the two-pass procedure removed it, and why; (3) change reviewer B's decision on "Cyber-physical attack graphs for substations" from X to I and report the new kappa; (4) explain in 100 words why one extra disagreement changes kappa more than raw agreement suggests. Expected evidence: outputs and explanation. Rubric: 3 = correct outputs, the ampersand failure diagnosed with a proposed fix, kappa recomputed correctly, chance-correction explained; 2 = one error; 1 = not run; 0 = none.

EX005 — Repair the eligibility criteria. Objective: turn disagreement into clearer criteria. Inputs: the three disagreements in Lab 1. Constraints: criteria must be decidable from a title and abstract or the first page of a report. Steps: (1) write inclusion and exclusion criteria for grey literature; (2) apply them to the three records; (3) state what you would do with a record you still cannot decide. Expected evidence: criteria and decisions. Rubric: 3 = criteria decidable, decisions consistent, undecidable records sent to full-text review rather than excluded; 2 = one decision inconsistent; 1 = criteria restate the topic; 0 = none.

EX006 — Concept matrix. Objective: synthesize by concept. Inputs: the three studies in §1.6 plus the NAO report on WannaCry and the NHS (October 2017). Constraints: read the NAO summary yourself; quote numbers with their source. Steps: (1) add the NAO report as a row; (2) add a column "rivals the study raises or leaves open"; (3) write a 200-word concept-centric paragraph on "regional effects of hospital cyber incidents". Expected evidence: matrix and paragraph. Rubric: 3 = NAO row accurate (scope of disruption and the recorded-versus-estimated cancellation distinction), rivals specific, paragraph organized by concept; 2 = paragraph author by author; 1 = numbers unsourced; 0 = none.

EX007 — Estimand and scope memo. Objective: define exactly what will be estimated. Inputs: RQ1 and RQ2. Constraints: the five-part template in §1.2. Steps: (1) write the estimand for RQ1; (2) write one for RQ2; (3) list two estimands a careless analyst might compute instead and why they differ; (4) state the hoped-for claim and the claims explicitly out of scope. Expected evidence: one page. Rubric: 3 = estimands unambiguous with a counterfactual contrast, alternatives genuinely different (for example first-day rise, rise versus last year), out-of-scope claims explicit; 2 = one estimand ambiguous; 1 = no contrast; 0 = none.

Chapter synthesis

A dissertation stands on its question. Move from topic to problem to question deliberately, making each narrowing defensible; test the question against FINER and write it in an infrastructure PICOT form with an observable exposure and a consequence outcome. Classify the question, because its type decides the design and the strongest claim available. Define the estimand before seeing data. Choose a review type that fits the project and say so. Search across disciplines and grey literature, test the search with known items, chain citations to a predeclared stopping point and log everything. Deduplicate, screen with a second reviewer on at least a sample, use kappa as a diagnostic of unclear criteria, and report the flow. Synthesize by concept: the concept matrix shows both what is known and the gap your study will fill.

Continue studying — chapter 01 of 08

← Course home · Course chapter index · Chapter 02 →

Continue to the next authored lesson.

Worked answers

EX001 — From topic to question. Problem: state drinking-water regulators and utility associations do not know how often cyber incidents at small utilities interrupt treatment or monitoring, or for how long, so support programmes cannot be targeted. Question (PICOT form): among community water systems serving 3,300–50,000 people in one state (P), how often did cyber incidents reported to the state primacy agency (E) coincide with a recorded treatment or monitoring interruption, compared with non-cyber equipment failures (C), measured as the number and duration of interruptions (O) over five years (T)? FINER: feasible only if the primacy agency's incident records distinguish causes and can be shared in aggregate — check this first; interesting to the agency and to utilities; novel if no state-level description exists; ethical because aggregate records involve no individuals, though small-utility identification risk must be handled (Chapter 3); relevant because it would show whether cyber causes are a meaningful share of interruptions. Hoped-for claim: "In one state over five years, recorded cyber incidents accounted for a stated share of treatment or monitoring interruptions, with stated durations." Out of scope: incidence of unreported incidents; health effects. Full credit requires honesty about feasibility (records may not code cause reliably) and a claim limited to reported incidents.

EX002 — Question types.

Q Type Minimum design Strongest claim
(a) Descriptive Complete county diversion log for 2024 with cause codes Hours of diversion by cause in that county and year
(b) Associational (causal only with more) Cross-sectional or panel comparison of EDs with and without drills, adjusted for size and case mix An association between drill adoption and waiting times during outages; causal only if adoption timing can be exploited (for example a staggered rollout) with credible comparisons
(c) Causal Controlled ITS or equivalent Percentage change relative to a counterfactual, under stated assumptions
(d) Explanatory Interviews with dispatchers and crews, dispatch records A mechanism supported by converging accounts
(e) Evaluative (causal) ITS around the protocol's start, ideally with a comparison region Effect of the protocol on time on diversion, under assumptions

The rubric's key point is (b): adoption of drills is chosen by hospitals, so better-run hospitals both adopt drills and have shorter waits; without a design that handles this, the claim is associational.

EX003 — Search strategy and known-item test. Blocks as in §1.4.2, with additions such as "health information technology failure", "cyber incident" and "EHR outage" in Block A and "regional surge", "patient redistribution" and "mutual aid" in Block C. Sources: a biomedical index (for emergency medicine and health services), a general scholarly index (for economics and public administration) and a computing index (for security), each justified by the discipline it covers. Grey literature: audit offices (for example the NAO), health agencies and ministries, regulators' statements, healthcare-coalition and EMS after-action reports, hospital-association surveys (for example the AHA's), and sector coordinating bodies. Known-item set: Dameff et al. (2023), Ghafur et al. (2019), Neprash et al. (2022); check by searching each source and confirming the item appears in the results, revising the string if not and logging the revision. Stopping rule: two consecutive chaining rounds with no new included study. Log fields: date, source, exact string, filters, hits, exported file name, person. The rubric rewards a search that would find the NAO report — impossible through bibliographic databases alone — and a known-item check that is actually executable.

EX004 — Deduplication and agreement. (1) The expected output as in §1.9: 20 identified, 5 duplicates, 15 screened, κ = 0.571. (2) With the ampersand record added and I appended for both reviewers, the output is records identified: 21 duplicates removed: 5 screened: 16, po=0.812 pe=0.547 kappa=0.586: the record was not removed. The normalized title "ransomware emergency department crowding at neighbouring hospitals" differs from "ransomware and emergency department crowding…" because the normalizer strips "&" but does not map it to "and". Fix: replace "&" with " and " before stripping punctuation, and add a manual review of near-matches (for example titles whose normalized forms differ by one word). (3) Starting from the original lab, changing B's decision on the substation paper to I gives po=0.733 pe=0.547 kappa=0.412. (4) Raw agreement fell by 6.7 points, but kappa fell from 0.571 to 0.412 because kappa rescales agreement above chance onto the remaining range: with chance agreement near 0.55, only 0.45 of the scale is "earned", so one extra disagreement among 15 records removes a large share of it.

EX005 — Repair the eligibility criteria. Criteria for grey literature: include reports by government bodies, regulators, audit offices, healthcare coalitions, EMS agencies or hospital associations that describe the operational effects (activity, diversion, delays, cancellations) of a specific cyber incident or IT outage on one or more health care organizations; exclude documents that describe procedures, guidance or products without reporting effects of an event, and vendor marketing. Decisions: the state health department memo on downtime procedures → exclude (procedure without effects) unless the full text reports an event's effects; the coalition exercise report → exclude for RQ1–2 (an exercise is not an event) but keep in a separate "methods and planning" list useful for RQ3 and the discussion; the vendor white paper → exclude. A record that remains undecidable from its first page goes to full-text review rather than being excluded, so that uncertainty does not silently reduce the evidence base.

EX006 — Concept matrix. NAO row: unit — NHS organizations in England; exposure — WannaCry, May 2017; comparison — none in the statistical sense; an investigation of what happened and why; outcomes — at least 81 of 236 trusts disrupted, plus 603 primary care and other NHS organizations (including 595 GP practices); 6,912 appointments recorded as cancelled and an estimated 19,000+ in total; design — audit investigation; supports — the scale of disruption and the gap between recorded and estimated cancellations; rivals raised or left open — incomplete recording; precautionary disconnection by unaffected organizations. Model paragraph: "Evidence on regional effects comes from three kinds of study. Audit investigations (NAO, 2017) establish the scale of disruption but rely on recorded counts that understate totals. Administrative-data analyses (Ghafur et al., 2019) quantify reduced activity at infected trusts — about 6% fewer admissions per infected hospital per day — and find no difference in deaths in A&E, a measure the authors call crude. A before-and-after study of neighbouring EDs (Dameff et al., 2023) associates an attack period with increased volume at unaffected neighbours. No study yet compares neighbours with an unaffected region or examines distance…" The rubric rewards the recorded-versus-estimated distinction and organization by concept.

EX007 — Estimand and scope memo. RQ1 estimand: "For each of Alder Point, Brookfield and Calloway: exp(β₂) − 1, the percentage difference between mean daily ambulance arrivals during days 0–20 and the arrivals expected from the ED's pre-event trend adjusted for the concurrent change in pooled Wexmoor arrivals." RQ2: "the pattern of these three estimates against distance, summarized as the difference between the nearest ED's estimate and the mean of the two more distant EDs' estimates, with intervals." Alternative estimands a careless analyst might compute: the first-day rise (captures the initial shock only); the rise relative to the same days last year (confounded by a year of trend and a different season); the total number of extra patients over 21 days (a different scale, sensitive to baseline size). Hoped-for claim as in §1.2.1; out of scope: patient outcomes, the effect of ransomware as distinct from diversion, other regions.