Chapter 18 — Research methods I
Lesson map
Lesson 18 published. Map, example, EX052.
flowchart TD
N0["Learning outcomes"]
N1["18.1 Research questions"]
N2["18.2 Methods"]
N3["18.3 Coding scheme"]
N4["18.4 Sampling"]
N5["18.5 Ethics"]
N6["18.6 Literature search"]
N7["Reliability between coders"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX052"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["18.1 Research questions"]
N2["18.2 Methods"]
N3["18.3 Coding scheme"]
N4["18.4 Sampling"]
N5["18.5 Ethics"]
N6["18.6 Literature search"]
N7["Reliability between coders"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX052"]
[Learning outcomes]
|
v
[18.1 Research questions]
|
v
[18.2 Methods]
|
v
[18.3 Coding scheme]
|
v
[18.4 Sampling]
|
v
[18.5 Ethics]
|
v
[18.6 Literature search]
|
v
[Reliability between coders]
|
v
[Worked example] --> [Exercise EX052]
Learning outcomes
You will (1) frame a researchable question; (2) choose qualitative or quantitative methods; (3) design a small coding scheme for reports; (4) consider ethics approval for research with people.
18.1 Research questions
"Is CTI useful?" is too broad. "Which HSSC report types led members to take a defensive action in 2026?" names a population, a period and an outcome. Good questions can be answered with available, lawful data.
Amara refines her own question several times with her supervisor before it is narrow enough. Each version is written down, so she can show how the question developed and why some broader versions were dropped. The final question can be answered from HSSC's feedback forms, which members have agreed may be used for research.
18.2 Methods
Quantitative methods count — how many reports mention phishing. Qualitative methods interpret — how analysts describe confidence. Many CTI studies combine both: code reports, count codes, then interview analysts.
Amara plans a mixed study: count report types and actions from the feedback forms, then interview a few analysts to understand why some types led to action. Each method covers a weakness of the other: counts show patterns, interviews explain them. Amara explains the choice in her research plan and lists what each method cannot show.
18.3 Coding scheme
Define categories in advance with examples: "entry route: phishing (T1566), exploited public application (T1190), valid account (T1078), unknown". Two coders label the same reports and compare; disagreements refine the definitions.
Amara writes the coding scheme with definitions and one example per code, and both coders practise on reports outside the study first. Agreement is measured before discussion, so it shows how clear the definitions are. Codes that cause repeated disagreement are rewritten or merged, and the changes are recorded with the reason.
18.4 Sampling
Public reports are a biased sample — vendors publish what is interesting or marketable. State the bias rather than pretending the sample represents all incidents.
Amara's study uses HSSC feedback forms, which have their own bias: members who fill them in may be more engaged than others. She states this and compares the number of responding members with the total. Readers can then judge how far the results may apply to the whole community and to other sectors with different sharing practices.
18.5 Ethics
Interviews with analysts need informed consent and, in a university, ethics review. Do not ask participants to reveal TLP-restricted content.
Amara's interview guide asks about practice in general terms, such as how analysts judge whether a report is relevant, without asking for report contents. Participants can stop at any time. Notes are anonymised and stored on university systems, and the ethics committee approves the plan before any interview takes place. HSSC's coordinator is informed but does not see individual answers.
18.6 Literature search
Before collecting data, search for prior work: peer-reviewed papers, standards and reputable practitioner reports. Record search terms and databases so the review can be repeated, and note when a source is a vendor blog rather than reviewed research.
Amara's search finds few studies on small sector communities. She records each search with the date, terms and databases, and lists the sources she included and excluded with reasons. Practitioner reports are kept but labelled, and their claims are checked against reviewed work where possible. The gap she finds in research on small communities becomes part of the justification for her study.
Reliability between coders
When two coders label twenty reports, count how often they agree and discuss each disagreement. Persistent disagreement on one category usually means its definition is vague — for example, whether a malicious link in a text message counts as phishing. Tighten the definition, add an example, and recode a fresh sample to check that agreement improved before analysing the full set.
Worked example
Amara's first question, "What do hackers do to hospitals?", is refined with her tutor to "Which initial-access techniques appear in public 2025–2026 incident reports about European health providers?" She drafts a four-category coding scheme and pilots it on five synthetic reports.
Synthetic case. The pilot reports are invented.
Errors and limits
Small samples cannot support strong claims. Report method limits alongside findings.
Exercises
- EX052 — Sharpen a question. Rewrite "Is CTI useful?" as a researchable question. Rubric: 3 = population, period, measurable outcome; 2 = two; 1 = still vague; 0 = none.
- EX053 — Public bias. Describe one bias in public incident reports. Rubric: 3 = selection—vendors publish notable/marketable cases, so counts skew; 2 = vague; 1 = minimal attempt; 0 = none.
- EX054 — Interview rule. What must Amara obtain before interviewing analysts? Rubric: 3 = informed consent and ethics review; avoid restricted content; 2 = consent only; 1 = vague; 0 = none.
Synthesis
Ask narrow questions, mix methods sensibly, code consistently, admit sampling bias and respect research ethics.
Continue studying — chapter 18 of 40
← Chapter 17 (draft) · Course chapter index · Chapter 19 (draft) →