Chapter 33 — Research methods II
Lesson map
Lesson 33 published. Map, example, EX065.
flowchart TD
N0["Learning outcomes"]
N1["33.1 Comparative design"]
N2["33.2 Uncertainty and replication"]
N3["33.3 Qualitative evidence"]
N4["33.4 Methods sections"]
N5["33.6 Practical activity"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX065"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["33.1 Comparative design"]
N2["33.2 Uncertainty and replication"]
N3["33.3 Qualitative evidence"]
N4["33.4 Methods sections"]
N5["33.6 Practical activity"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX065"]
[Learning outcomes]
|
v
[33.1 Comparative design]
|
v
[33.2 Uncertainty and replication]
|
v
[33.3 Qualitative evidence]
|
v
[33.4 Methods sections]
|
v
[33.6 Practical activity]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX065]
Learning outcomes
You will (1) design comparative studies with controlled variables; (2) handle measurement uncertainty and replication; (3) analyse qualitative data from interviews; and (4) write a methods section others could repeat.
33.1 Comparative design
Comparative questions — "which of two hybrid configurations performs better on branch links?" — need controlled variables: same hardware, same network emulation, same client versions, randomised order. Record everything that could differ. Prefer paired designs, where each condition runs on the same set of clients, so that client variation cancels out.
In a lab, small differences such as a background update on one virtual machine can be larger than the effect being measured. Run a short pilot first to check that the measurement itself is stable before collecting the main data.
33.2 Uncertainty and replication
Report medians and high percentiles with confidence intervals; repeat runs on different days; publish scripts and configurations. A result that disappears on replication was noise. For rare events such as handshake failures, plan sample sizes in advance so that confidence bounds are narrow enough to support the decision.
State the intervals in the write-up, not only in an appendix, so readers see how certain each result is. A difference smaller than the run-to-run variation should be reported as no detectable difference, which is itself a useful finding for a migration decision.
33.3 Qualitative evidence
Many migration questions are organisational: why did a team delay? what do vendors say about roadmaps? Semi-structured interviews, coded for recurring themes, answer such questions. Protect interviewees with consent, anonymisation and secure storage. Combine qualitative findings with measurements: interviews may explain why fallback persists at certain branches.
Interview notes are analysed with the same care as measurements: the coding scheme is written down, and a second person codes a sample to check agreement. Quotes are reported without names or details that would identify a branch or a person.
33.4 Methods sections
A methods section states question, design, setting, participants or systems, procedure, measures, analysis and limitations. The test is replication: could another student rerun your study from the text and obtain comparable results?
Write it as the study is done, not afterwards. Include the exact versions of software, the configuration of any network emulation, and how many runs were made. Limitations belong here too: for HCU studies, the most common is that the lab does not reproduce every device found in the branches.
33.5 Worked example
An HCU study compares ML-KEM-768 hybrid against classical X25519 handshakes on emulated branch links (40 ms RTT, 1 % loss). Paired design: 20 client VMs, each runs 500 handshakes per condition in random order, repeated on three days. Result (synthetic): median +0.8 ms (95 % CI 0.6–1.0), p99 +3.1 ms (CI 2.2–4.3). Interviews with five branch IT staff explain that two branches route through an old proxy, which the emulation did not include — a stated limitation and next study.
The proxy finding links to the rollout in Chapter 29. The study's scripts are kept so it can be rerun.
33.6 Practical activity
Write a methods section for a study of certificate-chain size effects on HCU's mobile app, including sample size reasoning.
Explain how many handshakes you need to detect a difference you care about, which client devices and network conditions you will emulate, and how you will separate the effect of chain size from other changes. State what result would lead HCU to keep ECDSA and what would support a move to ML-DSA. List the limitations you expect before collecting any data.
33.7 Common errors and limits
- Unpaired comparisons with uncontrolled clients.
- Reporting only means for skewed latency.
- Interviews without consent or anonymisation.
- Changing two variables at once, so that the effect cannot be attributed.
- Running all of one condition before the other, letting time-of-day effects creep in.
- Reporting one day's results as if they were general.
- Drawing conclusions from interviews with only the most enthusiastic staff.
Each weakens the evidence a migration decision depends on. The fix in each case is planning before data collection. Write the plan down first.
Exercises
- EX065 — Paired design. Explain why the paired design in 33.5 is stronger than running each condition on different clients. Rubric: 3 = variance reduction and confounding.
- EX066 — Methods checklist. List the eight elements of a methods section and give one HCU-specific example for two of them. Rubric: 3 = complete list with examples.
Synthesis
Controlled comparisons, replicated measurements with uncertainty, ethical qualitative inquiry and replicable methods sections make research on migration trustworthy.
Continue studying — chapter 33 of 40
← Chapter 32 (draft) · Course chapter index · Chapter 34 (draft) →