Chapter 3 — Study design, ethics and reproducibility | Master's Quantum and post-quantum security

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 03 published. Map, example, EX014.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["3.1 Designs"]
  N2["3.2 Defining the variables"]
  N3["3.3 Ethics and minimization"]
  N4["3.4 Rates and intervals"]
  N5["3.5 Reproducibility plan"]
  N6["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX014"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["3.1 Designs"]
  N2["3.2 Defining the variables"]
  N3["3.3 Ethics and minimization"]
  N4["3.4 Rates and intervals"]
  N5["3.5 Reproducibility plan"]
  N6["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX014"]

[Learning outcomes and connection]
    |
    v
[3.1 Designs]
    |
    v
[3.2 Defining the variables]
    |
    v
[3.3 Ethics and minimization]
    |
    v
[3.4 Rates and intervals]
    |
    v
[3.5 Reproducibility plan]
    |
    v
[Chapter synthesis]
    |
    v
[Worked example] --> [Exercise EX014]

Learning outcomes and connection

You will (1) choose between observational, quasi-experimental and testbed designs; (2) define outcome, exposure and unit of analysis; (3) minimize personal data in network logs; (4) estimate rates with honest intervals; and (5) plan a reproducible analysis. It operationalizes the question from Chapters 1–2 and prepares the competing explanations of Chapter 4.

3.1 Designs

An observational before/after study compares failure rates before and after the change; it is cheap but confounded by anything else that changed. A quasi-experiment exploits staggered rollout: if the hybrid group was enabled for half of managed browsers first, the other half is a concurrent control. A testbed reproduces handshakes through a copy of the proxy with and without the hybrid share and isolates the size mechanism directly, but cannot tell you how often the mechanism matters in real traffic. The Fragment Study combines a staggered-rollout quasi-experiment on LK-HS-1 with a testbed replication — each covers the other's weakness.

3.2 Defining the variables

Unit of analysis: a connection attempt series — the first attempt plus automatic retries within 10 seconds to the same host — so that retries do not inflate denominators. Exposure: whether the ClientHello offered X25519MLKEM768. Outcome: series ends without a completed handshake. Stratifiers: egress path (direct or legacy proxy), browser version, destination category. Write these definitions before seeing outcome data.

Store the definitions in the data dictionary with the date they were fixed. The pilot later changed one of them, defining a series by hostname rather than address, and that amendment is recorded alongside the original wording.

3.3 Ethics and minimization

Handshake logs contain client IP addresses and destination names, which can reveal health conditions (a hospital wing) or private behaviour. Minimize at collection: replace client addresses with a keyed hash rotated monthly; keep destination names only as category labels plus a salted hash; retain no payload; store raw logs only on university systems; obtain approval from the university's ethics or data-protection office; and consult the network team about what users were told. Interviews with help-desk staff (I01–I06) need informed consent and must avoid naming users.

3.4 Rates and intervals

Failure rates are small proportions, where the textbook normal interval misbehaves. The Wilson score interval behaves well near zero and is easy to compute with the standard library.

# Lab 3 (synthetic): Wilson 95% intervals for handshake failure rates before/after enabling hybrid.
import math
def wilson(k, n, z=1.96):
    p = k / n; d = 1 + z*z/n
    c = (p + z*z/(2*n)) / d; h = z*math.sqrt(p*(1-p)/n + z*z/(4*n*n)) / d
    return c - h, c + h
for label, k, n in [("before (x25519 only)", 18, 40000), ("after (hybrid offered)", 61, 40000)]:
    lo, hi = wilson(k, n)
    print(f"{label:24s} {k:3d}/{n} = {1e4*k/n:5.1f} per 10k  95% CI [{1e4*lo:4.1f}, {1e4*hi:4.1f}]")

Expected output:

before (x25519 only)      18/40000 =   4.5 per 10k  95% CI [ 2.8,  7.1]
after (hybrid offered)    61/40000 =  15.2 per 10k  95% CI [11.9, 19.6]

The synthetic increase from 4.5 to 15.2 failures per 10,000 series is clear — the intervals do not overlap — but it says nothing yet about why. That is Chapter 4.

The lab computes the rates for the whole corpus, before stratification. Because both rates are small, the intervals are asymmetric, which the Wilson method handles correctly. Report counts and denominators with every rate, so a reader can recompute the interval; 18 and 61 failures out of 40,000 series each are the figures that later split by path in Lab 4.

3.5 Reproducibility plan

Pin the analysis to versioned code and a data dictionary; generate every number in the dissertation from scripts; store the minimized dataset with a checksum; record the exact browser policy that enabled the group and its date. Lab 3b below also documents the size assumptions behind the mechanism so that a reader can recompute them.

# Lab 3b: key-share sizes in a TLS 1.3 ClientHello (sizes from FIPS 203 and RFC 10024).
# X25519 share = 32 bytes; ML-KEM-768 encapsulation key = 1184 bytes; ciphertext = 1088 bytes.
BASE_HELLO = 520          # synthetic size of the rest of a typical ClientHello (illustrative)
MSS = 1460                # TCP payload per segment on a 1500-byte Ethernet path, no options
shares = {"x25519 only": 32,
          "X25519MLKEM768": 1184 + 32,
          "X25519MLKEM768 + x25519": 1184 + 32 + 32}
for name, size in shares.items():
    hello = BASE_HELLO + size + 4          # 4 bytes: group id + length per share (approx.)
    segs = -(-hello // MSS)
    print(f"{name:26s} ClientHello ~{hello:5d} B -> {segs} TCP segment(s)")
print("server share for X25519MLKEM768:", 1088 + 32, "bytes")

Expected output:

x25519 only                ClientHello ~  556 B -> 1 TCP segment(s)
X25519MLKEM768             ClientHello ~ 1740 B -> 2 TCP segment(s)
X25519MLKEM768 + x25519    ClientHello ~ 1772 B -> 2 TCP segment(s)
server share for X25519MLKEM768: 1120 bytes

The rest-of-ClientHello size (520 bytes) is an illustrative assumption; real sizes vary with extensions, server name length and padding, which is why the testbed measures them directly.

3.6 Common errors and limits

Counting retries as independent attempts.
Collecting full destination names "just in case".
Using normal-approximation intervals for rates near zero.
Treating a before/after difference as a causal effect.

Each error has a guard already in the design: series instead of attempts as the unit, destination categories plus salted hashes instead of names, Wilson intervals for small rates, and a staggered rollout with a concurrent control to separate the effect of the hybrid group from anything else that changed during the same weeks.

Exercises — chapter 3

EX014 — Design choice. Compare the three designs for the Fragment Study in a table of strengths and threats. Rubric: 3 = each threat matched to a design that addresses it.
EX015 — Unit of analysis. Explain how retries could inflate the failure denominator and bias the rate. Rubric: 3 = direction of bias stated.
EX016 — Minimization plan. Write the data-minimization section for the ethics application. Rubric: 3 = keyed hashing, categories, retention and access covered.
EX017 — Wilson by hand. Compute the Wilson interval for 9 failures in 20,000 series with Lab 3. Rubric: 3 = correct output and interpretation.
EX018 — Size sensitivity. Change Lab 3b's base size to 300 and 1,300 bytes; when does the x25519-only hello need two segments? Rubric: 3 = correct runs and a conclusion about why the testbed must measure.
EX019 — Reproducibility checklist. List eight items a reader needs to reproduce your numbers. Rubric: 3 = includes policy date and data checksum.

Chapter synthesis

A staggered-rollout quasi-experiment plus a testbed replication, with pre-defined units and outcomes, minimized logs, Wilson intervals and scripted numbers, gives a design whose results can be both trusted and rerun.

Continue studying — chapter 03 of 08

← Chapter 02 (draft) · Course chapter index · Chapter 04 (draft) →