Chapter 18 — Research methods I | Bachelor's Incident response and cyber resilience

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 18 published. Map, example, EX052.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["18.1 From practical question to research question"]
  N2["18.2 Variables"]
  N3["18.3 Baselines and comparisons"]
  N4["18.4 Validity"]
  N5["18.5 Reproducibility"]
  N6["18.6 Ethics"]
  N7["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX052"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["18.1 From practical question to research question"]
  N2["18.2 Variables"]
  N3["18.3 Baselines and comparisons"]
  N4["18.4 Validity"]
  N5["18.5 Reproducibility"]
  N6["18.6 Ethics"]
  N7["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX052"]

[Learning outcomes and connection]
    |
    v
[18.1 From practical question to research question]
    |
    v
[18.2 Variables]
    |
    v
[18.3 Baselines and comparisons]
    |
    v
[18.4 Validity]
    |
    v
[18.5 Reproducibility]
    |
    v
[18.6 Ethics]
    |
    v
[Chapter synthesis]
    |
    v
[Worked example] --> [Exercise EX052]

Learning outcomes and connection

You will be able to (1) turn a practical question into a research question with defined variables; (2) choose a baseline and comparison; (3) identify threats to internal and external validity; (4) plan for reproducibility; and (5) recognize ethical constraints on studying response work. Chapters 12 and 15 gave you statistics and measurement; this chapter organizes them into a study. Chapter 33 deepens the method for outcome measurement, and Chapter 35 asks you to propose a capstone.

18.1 From practical question to research question

"Is playbook B better?" is not yet answerable. A research question names the population (which cases), the intervention and comparison (playbook B vs playbook A), the outcome (for example correct containment decision within 30 minutes) and the setting (simulated cases, trained student analysts). Each element limits what the result will mean.

Writing the elements out also exposes disagreement early. A manager asking whether playbook B is better may mean faster triage; an instructor may mean more correct escalations. The study in the worked example chooses correct escalation decisions as its main outcome and time per case as secondary, and says so before any data is collected.

18.2 Variables

The independent variable is what you vary (playbook). The dependent variable is what you measure (decision correctness, time). Confounders affect both assignment and outcome (analyst experience, case difficulty). Control variables are held constant or measured and adjusted for.

In the playbook study, the independent variable is the playbook, the dependent variable is the proportion of correct escalations, and analyst ability and case difficulty are confounders. Ability is handled by having every analyst use both playbooks; difficulty is handled by matching two sets of cases in a pilot. Naming each variable this way makes clear which threat each design choice is meant to address.

18.3 Baselines and comparisons

Without a baseline you cannot say "better". The comparison can be between groups (some analysts use A, others B), within subjects (each analyst uses both, on different but matched cases, in balanced order) or before/after (weakest, because other things change over time). Randomization of assignment is the strongest protection against confounding that you cannot measure.

The playbook study chooses a within-subjects comparison with counterbalanced order, so each of the 24 analysts acts as their own baseline. Comparing each analyst's results with both playbooks removes differences in ability from the comparison, while the balanced order spreads any learning effect evenly across A and B.

18.4 Validity

Internal validity: does the design support the causal claim within the study? Threats: confounding, learning effects, unequal case difficulty, measurement differences. Construct validity: does "correct decision" measure what matters? External validity: would results hold for real incidents, experienced staff, other organizations? Simulated cases help internal validity and hurt external validity; say so.

The playbook study states its own threats openly: student analysts limit external validity, the instructors' reference answers may be imperfect, and being observed may change behaviour. Listing them does not weaken the study; it tells a reader exactly how far the conclusion can travel and which follow-up study would extend it.

18.5 Reproducibility

Pre-register or at least write down the question, design, outcome and analysis before collecting data; keep data and code together; record versions; publish the synthetic materials so others can repeat the study.

For the playbook study, that means saving the two case sets, the reference answers set by the panel, the order assignment for each analyst and the analysis script, with dates and versions. Another instructor should be able to rerun the comparison from those files alone. Note any deviation from the written plan, such as an analyst who withdrew, and how it was handled in the analysis.

18.6 Ethics

Studies of response work involve people (analysts) whose performance is measured, and possibly incident data about others. Use synthetic cases where possible; get informed consent; ensure results cannot be used to evaluate individuals; obtain review from an ethics committee where your institution requires it. The Menlo Report (Chapter 10) adapts research-ethics principles to ICT.

In the playbook study, results for individual analysts must not reach anyone grading them, and any analyst may withdraw without consequence. Consent forms should explain what is recorded, who sees it and how long it is kept. Where the analysts are students on a course, keep the study separate from assessment so that taking part, or declining, cannot affect a mark.

18.7 Worked example (synthetic): comparing two triage playbooks

Fictional proposal. Question: "Among 24 student analysts triaging 20 synthetic phishing cases, does playbook B (checklist with impact questions) produce more correct escalation decisions than playbook A (free-form guidance)?"

Design: within-subjects; each analyst triages 10 cases with A and 10 with B; two matched case sets (Set 1, Set 2) built to equal difficulty by a pilot; order counterbalanced (half do A-then-B, half B-then-A; set assignment balanced). Outcome: proportion of correct escalation decisions, with "correct" defined in advance by a panel of three instructors who did not see the analysts' answers. Secondary: time per case.

Confounders addressed: analyst ability (within-subjects), learning (counterbalanced order), case difficulty (matched sets, balanced assignment). Remaining threats: students are not professionals (external validity); the panel's reference answers may be imperfect (construct validity); analysts know they are observed (Hawthorne effect).

Defensible conclusion if B scores higher: "In this simulation with student analysts, playbook B was associated with more correct escalations"; not "playbook B improves incident response".

18.8 Common errors and limits

Before/after comparisons presented as causal.
Defining "correct" after seeing results.
Using the same people to design cases, train analysts and score answers.
Overgeneralizing from simulations or small samples.
Building matched case sets but never checking in a pilot that they really are equally difficult.
Measuring only the outcome that the preferred option is expected to win.
Dropping participants who withdrew without reporting how many and why, which can bias the comparison in either direction. A further limit: a significant difference in a simulation answers only the narrow question asked.

Exercises — chapter 18

Attempt each exercise before opening the worked results. Every case, organization, person, address and number in an exercise is synthetic unless a cited public source is named.

EX052 — Structure the question. Inputs: "Do daily stand-ups make our response faster?" Steps: write population, intervention, comparison, outcome, setting; name two confounders. Rubric: 3 = all five defined, measurable outcome, confounders such as case volume and staffing; 2 = one vague; 1 = still a slogan; 0 = none.

EX053 — Critique a design. Inputs: a team adopts playbook B in April and reports that correct escalations rose from 70% in March to 82% in April. Steps: list three alternative explanations. Rubric: 3 = case mix change, learning over time, measurement change or new staff; 2 = two; 1 = one; 0 = none.

EX054 — Ethics checklist. Inputs: §18.7. Steps: write five ethics safeguards. Rubric: 3 = consent with right to withdraw, synthetic cases only, no individual results to graders affecting course marks (or separated), pseudonymized data, institutional review; 2 = three; 1 = generic; 0 = none.

Chapter synthesis

A study is a structured comparison: define population, intervention, comparison, outcome and setting; protect internal validity with randomization and balance; state external limits; fix definitions before data; and protect participants.