Chapter 33 — Research methods II
Lesson map
Lesson 33 published. Map, example, EX065.
flowchart TD
N0["Learning outcomes and connection"]
N1["33.1 Qualitative methods"]
N2["33.2 Coding and agreement"]
N3["33.3 Reproducible quantitative analysis"]
N4["33.4 Mixed methods"]
N5["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX065"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["33.1 Qualitative methods"]
N2["33.2 Coding and agreement"]
N3["33.3 Reproducible quantitative analysis"]
N4["33.4 Mixed methods"]
N5["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX065"]
[Learning outcomes and connection]
|
v
[33.1 Qualitative methods]
|
v
[33.2 Coding and agreement]
|
v
[33.3 Reproducible quantitative analysis]
|
v
[33.4 Mixed methods]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX065]
Learning outcomes and connection
After this chapter you can (1) design interviews and document analysis as qualitative methods, including ethical approval and consent; (2) code qualitative data with a codebook and check agreement between coders; (3) make a quantitative analysis reproducible; and (4) combine qualitative and quantitative evidence in a mixed-methods design. It builds on Chapter 18 and prepares the comparative project (Chapter 34) and the capstone (Chapters 35–40).
33.1 Qualitative methods
Many important questions in critical-sector security are about people and organisations: why operators bypass a control, how incident decisions were made, what practitioners believe works. Semi-structured interviews follow a guide of open questions while allowing follow-up. Document analysis examines plans, reports and policies. Both require ethical approval from your institution when they involve people, informed consent (participants know the purpose, what will be recorded and how it will be stored, and can withdraw), and protection of participants and their organisations: no detail that identifies a person or exposes a vulnerability of a real operator should appear in your thesis. Interview notes about security weaknesses are sensitive data; store them encrypted and delete them on schedule (Chapter 14).
33.2 Coding and agreement
Coding assigns labels to passages: "fallback to paper", "vendor dependency", "unclear decision owner". A codebook defines each code with inclusion and exclusion rules and an example. To check reliability, two people code the same sample independently and measure agreement. Raw percentage agreement overstates reliability because some agreement happens by chance; Cohen's kappa corrects for this: $\kappa = (p_o - p_e)/(1 - p_e)$, where $p_o$ is observed agreement and $p_e$ the agreement expected by chance from each coder's label frequencies. Disagreements are not failures; discussing them sharpens the codebook.
33.3 Reproducible quantitative analysis
A reproducible analysis keeps raw data unchanged, does all transformations in scripts, records versions and random seeds, and produces every number in the report from code (Sandve et al., 2013). The labs in this book follow these rules: fixed seeds, standard library only, printed outputs reproduced in the text. If a simulation or model is used, state how it was checked against reality — verification and validation of simulation models is its own discipline (Sargent, 2013).
33.4 Mixed methods
Quantitative evidence says how much; qualitative evidence says how and why. A mixed-methods design might count, across 30 public incident reports, how many mention a fallback to manual operation (quantitative), then interview four operators about what makes manual fallback work (qualitative). The combination is stronger than either, provided each part is done properly and the integration — how one informs the other — is explicit.
In the 30-report design, the link is fixed before collection: any fallback condition that appears in at least five reports becomes an interview question.
In the 30-report example, the count tells you how often reports mention manual fallback, and the interviews explain why fallback sometimes fails, for example because staff have not practised it. Integration means the interview guide asks about the conditions the count revealed.
33.5 Worked example: coding incident reports (synthetic data)
The data are synthetic. Two students code 20 incident-report excerpts for the presence of the code "unclear decision owner". They agree on 16 of 20 ($p_o = 0.80$). Student A applied the code to 8 excerpts; student B to 6. Chance agreement: both "yes" $= (8/20)(6/20) = 0.12$; both "no" $= (12/20)(14/20) = 0.42$; $p_e = 0.54$. Kappa $= (0.80 - 0.54)/(1 - 0.54) = 0.26/0.46 \approx 0.57$ — moderate agreement. The students review the four disagreements, find that one coded "decision delayed" as "owner unclear" even when the owner was named, and tighten the definition: "applies only when the report states or shows that participants did not know who held the decision".
33.6 Common errors and limits
- Interviewing practitioners without ethics approval or consent.
- Publishing identifiable details of real operators' weaknesses.
- Reporting percentage agreement as reliability.
- Analysis steps done by hand and not recorded.
- "Mixed methods" that are two separate studies never integrated.
In the coding example, reporting 80 % agreement instead of kappa 0.57 would be the third error. Interviewing CRP engineers about remote access without consent and approval would be the first, and quoting a named operator's unpatched gateway would be the second. A spreadsheet edited by hand without a record would be the fourth.
Exercises — chapter 33
EX065 — Compute kappa. Objective: measure reliability. Inputs: two coders, 40 excerpts, agree on 34; coder A applied the code 10 times, coder B 12 times. Steps: compute $p_o$, $p_e$ and $\kappa$, and interpret. Expected evidence: working and interpretation. Rubric: 3 = $p_o = 0.85$; $p_e = 0.25 \times 0.30 + 0.75 \times 0.70 = 0.075 + 0.525 = 0.60$; $\kappa = 0.625$, substantial-to-moderate agreement with a note that interpretation scales are conventions; 2 = arithmetic slip; 1 = $p_o$ only; 0 = none.
EX066 — Consent and protection. Objective: design ethical qualitative work. Inputs: plan to interview three CRP operators' engineers about remote-access practices. Steps: write the consent information (five points) and three data-protection measures. Expected evidence: lists. Rubric: 3 = purpose, voluntary participation/withdrawal, what is recorded, storage/retention, how results are anonymised, plus encryption, pseudonyms, no vulnerability details in outputs/host review; 2 = partial; 1 = generic; 0 = none.
Chapter synthesis
Qualitative methods answer how and why; they need ethics approval, consent and protection of participants and organisations. Codebooks and kappa make coding reliable; scripts, seeds and versions make quantitative analysis reproducible; mixed methods are only as good as their integration.
Continue studying — chapter 33 of 40
← Chapter 32 · Course chapter index · Chapter 34 →
Continue to the next authored lesson.