Chapter 18 — Research methods I

Chapter 18 — Research methods I

Lesson map

Lesson 18 published. Map, example, EX035.

flowchart TD
  N0["Learning outcomes"]
  N1["18.1 Questions"]
  N2["18.2 Methods"]
  N3["18.3 Populations, samples and validity"]
  N4["18.5 Practical activity"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX035"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["18.1 Questions"]
  N2["18.2 Methods"]
  N3["18.3 Populations, samples and validity"]
  N4["18.5 Practical activity"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX035"]

[Learning outcomes]
    |
    v
[18.1 Questions]
    |
    v
[18.2 Methods]
    |
    v
[18.3 Populations, samples and validity]
    |
    v
[18.5 Practical activity]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX035]

Learning outcomes

You will (1) formulate a researchable question; (2) choose between literature review, measurement, experiment and case study; (3) design a simple measurement study with a stated population and sample; and (4) identify threats to validity.

18.1 Questions

A good research question is specific, answerable with available evidence and relevant to a decision. "Is PQC good?" is not researchable. "What share of HCU's branch-to-data-centre flows carry data with a confidentiality lifetime above 15 years?" is. Questions in this field often take the forms how much (measurement), what happens if (experiment), how did it go (case study) or what is known (literature review).

HCU's security office asks the student team for questions of the second kind, because each answer changes a migration priority or a budget line.

18.2 Methods

A literature review summarises primary sources systematically, recording search terms and inclusion rules. A measurement study counts properties of a population, such as TLS groups offered by a set of servers. An experiment changes one factor and observes the effect, such as handshake latency with and without a hybrid group in a lab. A case study examines one migration in depth. Each method answers different questions; combining them strengthens conclusions.

HCU's own studies combine a measurement of its estate with lab experiments on the slowest branch links.

18.3 Populations, samples and validity

State the population (all HCU web services), the sampling frame (services in the asset register) and the sample (those reachable from the test network). Internal validity asks whether the observed effect is caused by the factor studied; external validity asks whether results generalise. A latency experiment on a fast lab network may not predict results on rural branch links; that is an external-validity limit, not a failure, as long as it is stated.

HCU's asset register itself may be incomplete, which is a frame problem to state.

18.4 Worked example

Question: "Does enabling X25519MLKEM768 on HCU's test gateway increase median handshake time for branch clients?" Method: lab experiment with emulated branch latency of 40 ms and 1 % loss; 1,000 handshakes per condition, alternating order to control drift. Result (synthetic): median increases by 0.9 ms; the 99th percentile increases by 3 ms because some ClientHellos need a second segment. Validity note: emulated loss does not reproduce the behaviour of the branch's real inspection proxy.

The report states both the median and the tail, and the limit of the emulation. HCU's network team then tested at one branch with the real proxy before wider rollout, which is where the external-validity gap would show.

18.5 Practical activity

Write a one-page study protocol for measuring how many HCU internal services support TLS 1.3, including population, frame, sample, method, data handling and validity threats.

A protocol written before collecting data stops later choices from being shaped by the results. Say how you will count a service that does not respond, how you will treat services outside the register and how long the data will be kept. Validity threats might include services only reachable from some networks or a test network that differs from production. Have a classmate read the protocol and try to find a choice you left unstated.

18.6 Common errors and limits

• Questions too broad to answer.
• Unstated sampling frames that hide selection bias.
• Generalising lab latency to production networks.

A broad question produces vague answers that no one can act on. An unstated frame hides which systems were never in the sample, and the missing ones are often the oldest and most fragile. Lab networks are cleaner than production, so latency and failure results should be presented as lower bounds until confirmed in the field. Write each limit next to the result it affects, not in a separate place readers may skip.

Exercises

• EX035 — Sharpen the question. Turn "Are our vendors ready for quantum?" into a researchable question with a method. Rubric: 3 = specific, measurable, decision-relevant.
• EX036 — Validity. Name one internal and one external validity threat in the experiment of 18.4. Rubric: 3 = both correct and explained.

Synthesis

Researchable questions, explicit populations and stated validity limits turn opinions about quantum readiness into evidence.

Continue studying — chapter 18 of 40

← Chapter 17 (draft) · Course chapter index · Chapter 19 (draft) →