Chapter 18 — Research methods I
Lesson map
Lesson 18 published. Map, example, EX035.
flowchart TD
N0["Learning outcomes and connection"]
N1["18.1 From topic to question"]
N2["18.2 Designs"]
N3["18.3 Literature, systematically"]
N4["18.4 Validity"]
N5["18.6 Practical activity"]
N6["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX035"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["18.1 From topic to question"]
N2["18.2 Designs"]
N3["18.3 Literature, systematically"]
N4["18.4 Validity"]
N5["18.6 Practical activity"]
N6["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX035"]
[Learning outcomes and connection]
|
v
[18.1 From topic to question]
|
v
[18.2 Designs]
|
v
[18.3 Literature, systematically]
|
v
[18.4 Validity]
|
v
[18.6 Practical activity]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX035]
Learning outcomes and connection
After this chapter you can (1) turn a topic into a researchable question; (2) choose between descriptive, comparative, experimental and case-study designs; (3) find, screen and summarise literature systematically and record how; and (4) identify threats to validity in a security study. It builds on Chapters 4, 12 and 15 and prepares Chapter 23 (risk methods), Chapter 33 (research methods II) and the capstone proposal (Chapter 35).
18.1 From topic to question
"Ransomware in hospitals" is a topic. A research question is specific enough that evidence could answer it and someone could disagree: "In publicly reported ransomware incidents affecting hospitals in 2017–2024, how often did reports describe a fallback to paper processes, and what limits of that fallback did they mention?" A good question names the population (which hospitals, which incidents), the variables (fallback described, limits mentioned), the time frame and the kind of answer (a proportion and themes). At bachelor's level, questions that compare — options, sectors, methods — fit the course's outcome better than questions that claim to discover something new.
18.2 Designs
Descriptive studies say what is (an inventory survey; a count of advisories by sector). Comparative studies contrast groups or options using the same criteria. Experimental studies change one thing deliberately and measure the effect, ideally against a control (the detector threshold experiment of Chapter 27). Case studies examine one or a few cases in depth, using several sources; they explain how and why, but generalise cautiously. Choose the design that the question needs, not the one that seems most scientific.
For the CEDC backup question, comparative is the fitting design: three strategies scored on the same criteria. An experiment would compare drills under controlled conditions, and a case study would examine one past restore in depth. Each answers a different question, so choose the design after writing the question.
18.3 Literature, systematically
Searching for sources is itself a method and should be recorded: databases and sites searched, search terms, date, inclusion and exclusion criteria, numbers found and kept. Distinguish primary sources (an agency advisory, a regulator's report, a peer-reviewed study) from secondary (a news article or blog summarising them). Prefer primary, cite what you read (Chapter 4), and record why you excluded what you excluded so that someone else could repeat the screening. A simple spreadsheet — source, type, date, relevance, key finding, limits — is enough.
18.4 Validity
Four classic threats. Construct validity: does the measure capture the concept? (Counting "incidents reported" measures reporting, not incidents.) Internal validity: is the claimed cause really the cause? (Hospitals with more security staff may report more incidents because they detect more.) External validity: does the result generalise? (Findings from large urban hospitals may not apply to rural clinics.) Reliability: would another researcher get the same result? (Two coders classifying incident reports should agree; measure their agreement.) Sommer and Paxson's critique of machine-learning intrusion detection is a classic example of external-validity problems in security research (Sommer and Paxson, 2010).
18.5 Worked example: a comparative question (synthetic plan)
This plan is synthetic. A student in the CRP's internship programme asks: "Which of three backup strategies best supports four-hour restoration of CEDC's dispatch service?" Design: comparative, using the same criteria (restore time measured in an isolated drill, resilience to backup-server compromise, operational burden, cost). Data: three timed restores in an isolated test environment provided by CEDC under written authorisation, vendor documentation, and interviews with two administrators. Validity notes: three drills are a small sample (report intervals, Chapter 37); the test environment may differ from production (external validity — document differences); the student designed the drills (reliability — have an administrator repeat one). Output: a recommendation with stated assumptions, not a universal claim about backup products.
18.6 Practical activity
Write three research questions on one topic of your choice in critical-sector security: one descriptive, one comparative and one case-study question. For each, name the population, variables, time frame and the type of evidence you would need.
For example, on unsupported medical devices: descriptive, "how many CVH clinical devices were past end of support in 2026?"; comparative, "which of three approaches reduces exposure most within a budget?"; case study, "how did one hospital manage its imaging workstations during a replacement delay?" Note for each whether the evidence is obtainable under authorisation.
18.7 Common errors and limits
- Questions too broad to answer or too narrow to matter.
- Unrecorded literature searches that cannot be repeated.
- Over-generalising from one case.
- Mistaking correlation in incident data for cause.
- Collecting data about people or real systems without ethical approval and authorisation (Chapter 10).
The CEDC backup plan avoids these by naming the population (CEDC's dispatch service), limiting the claim to three strategies, recording sources, and treating three drills as a small sample. A study of "ransomware in hospitals" would fail the first test. A student who interviewed CEDC administrators without consent would fail the fifth whatever the quality of the analysis.
Exercises — chapter 18
EX035 — Sharpen the question. Objective: make a topic researchable. Inputs: topic "phishing in the energy sector". Steps: write a research question with population, variables, time frame and answer type; state which design fits. Expected evidence: question and design. Rubric: 3 = specific, answerable with obtainable evidence, design justified; 2 = specific but evidence unobtainable; 1 = still a topic; 0 = none.
EX036 — Name the threat. Objective: identify validity threats. Inputs: "Hospitals that bought product X had 40% fewer reported ransomware incidents, so X prevents ransomware." Steps: name and explain three validity threats. Expected evidence: three threats. Rubric: 3 = construct (reported vs actual), internal (confounders such as budget/maturity, selection), external (which hospitals), or reliability; 2 = two; 1 = one; 0 = none.
Chapter synthesis
Research starts with a question specific enough to be answered and disputed. Choose the design the question needs, search and screen literature in a way someone else could repeat, and name the threats to validity before a reviewer does.
Continue studying — chapter 18 of 40
← Chapter 17 · Course chapter index · Chapter 19 →
Continue to the next authored lesson.