Chapter 24 — Applied research methods | Technologist Quantum and post-quantum security

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 24 published. Map, example, EX047.

flowchart TD
  N0["Learning outcomes"]
  N1["24.1 Practical questions"]
  N2["24.2 Experiments"]
  N3["24.3 Measurement studies"]
  N4["24.4 Reporting"]
  N5["24.6 Practical activity"]
  N6["24.7 Sample sizes"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX047"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["24.1 Practical questions"]
  N2["24.2 Experiments"]
  N3["24.3 Measurement studies"]
  N4["24.4 Reporting"]
  N5["24.6 Practical activity"]
  N6["24.7 Sample sizes"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX047"]

[Learning outcomes]
    |
    v
[24.1 Practical questions]
    |
    v
[24.2 Experiments]
    |
    v
[24.3 Measurement studies]
    |
    v
[24.4 Reporting]
    |
    v
[24.6 Practical activity]
    |
    v
[24.7 Sample sizes]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX047]

Learning outcomes

You will (1) frame practical research questions for implementation teams; (2) design small experiments and measurement studies; (3) control variables and repeat runs; and (4) report results honestly.

24.1 Practical questions

Implementation research answers questions like: does the hybrid group increase failures on the warehouse network? Which partner clients negotiate hybrid SSH? How much does an ML-DSA-44 chain add to mesh latency? Each has a population, a measure and a decision.

Writing the decision first keeps the study small. For the warehouse question KLS decided in advance: if the hybrid failure rate's upper bound exceeds the classical one by more than the 0.04-point budget, warehouse sites get a separate listener until proxies are upgraded; otherwise the profile goes ahead unchanged.

24.2 Experiments

Change one variable (hybrid on/off, chain type), hold others fixed (hardware, network emulation, client versions), randomise order, repeat runs, and record everything. Paired designs, where each client runs both conditions, reduce noise.

KLS's lab uses network emulation to fix loss and delay, pins client library versions by image digest, and lets a script choose the order of conditions from a recorded seed. Each run logs its configuration hash, so a later reader can tell exactly which condition produced each row. A paired design for the warehouse question runs every client type under hybrid and classical within the same hour, so time-of-day load affects both conditions equally.

24.3 Measurement studies

Count properties across a population: groups negotiated by partner SFTP clients over a month, image library versions across the registry. Define the population and coverage; report unknowns.

For the partner study KLS defined the population as every partner account with at least one SFTP session in a 30-day window, counted 23 accounts, and reported the KEX algorithm per account from sshd logs. Two dormant partners had no session and were listed as unknown rather than dropped, since they could reconnect with an old client at any time.

24.4 Reporting

Report medians and percentiles with intervals, numbers of runs, environment and limitations. Share scripts. If a result surprises you, replicate before acting.

KLS's report template has fixed headings: question, population or setup, number of runs, results with intervals, environment, limitations and decision. Latency is reported as median and 99th percentile, never as a mean, because handshake times are skewed by a few slow clients. The scripts and raw data are linked from the report, and a surprising result is rerun on a rebuilt lab before anyone acts on it.

24.5 Worked example

Question: does ML-DSA-44 mTLS increase p99 latency for mesh calls? Design: staging cluster, 10 service pairs, 2,000 calls per pair per condition, randomised order, three days. Result (synthetic): p50 +0.3 ms, p99 +1.9 ms (95 % CI 1.2–2.7); CPU unchanged within noise. Limitation: staging traffic patterns are simpler than production. Decision: proceed to a production canary.

The design choices each remove a confounder. Ten service pairs cover different payload sizes; randomised order spreads time-of-day effects across conditions; three days cover the daily batch pattern. The confidence interval for p99 excludes zero, so the increase is real but small, and the limitation is stated where the decision is.

24.6 Practical activity

Design an experiment comparing hybrid and classical handshake failure rates on an emulated warehouse network with 2 % loss.

Specify the emulated network (2 % loss, a fixed delay such as 80 ms, one MTU), the client library versions, the two conditions (hybrid offered and preferred; classical only), the number of handshakes per condition and how you chose it, and the order in which runs execute. Use a paired design so each client runs both conditions. State the decision rule in advance, for example comparing 95 % upper bounds of the failure rates with a budget, and list one limitation of emulation compared with real warehouse links.

24.7 Sample sizes

For rare events, sample size decides whether a result is informative. To show a failure rate below 0.1 % with reasonable confidence, you need several thousand trials with few failures; with a few hundred, even zero failures leaves an upper bound near 1 %. Estimate the needed sample before starting, using the confidence-bound method of Chapter 19, and plan the run length accordingly rather than stopping when the result looks good.

KLS planned the warehouse study for 6,000 handshakes per condition for that reason.

With 6,000 handshakes and no failures, the upper bound is about 0.05 %, below the 0.1 % target; at 600, the same clean result would only have bounded the rate near 0.5 %.

24.8 Common errors and limits

Changing several variables at once.
Stopping early when results look favourable.
Reporting means of skewed latency.

KLS's first warehouse test changed the client library and the group list together, so its result could not say which change mattered. A canary was once stopped after a good first hour, before the evening batch that usually exposed failures. And an early report quoted mean handshake latency, which a handful of slow scanner connections pulled up by several milliseconds; the median and 99th percentile told a clearer story. Each error makes a result look more decisive than the evidence allows.

Exercises

EX047 — Design check. Identify two weaknesses in an experiment that tests hybrid on Monday and classical on Friday. Rubric: 3 = time confounding, unpaired/load differences.
EX048 — Zero failures. With 300 trials and 0 failures, roughly what upper bound on the failure rate can you claim (rule of three)? Rubric: 3 = ≈ 1 % (3/300).

Synthesis

Practical research uses controlled, repeated experiments and well-defined measurement studies with honest reporting and sample sizes planned in advance.

Continue studying — chapter 24 of 30

← Chapter 23 (draft) · Course chapter index · Chapter 25 (draft) →