Chapter 6 — Quantum and post-quantum security: controlled evaluation
Lesson map
Lesson 06 published. Map, example, EX026.
flowchart TD
N0["Learning outcomes"]
N1["6.1 Evaluation questions"]
N2["6.2 Thresholds before data"]
N3["6.3 Practical activity — Lab K"]
N4["6.4 Interoperability matrices"]
N5["6.5 Validity"]
N6["6.6 Worked case"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX026"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["6.1 Evaluation questions"]
N2["6.2 Thresholds before data"]
N3["6.3 Practical activity — Lab K"]
N4["6.4 Interoperability matrices"]
N5["6.5 Validity"]
N6["6.6 Worked case"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX026"]
[Learning outcomes]
|
v
[6.1 Evaluation questions]
|
v
[6.2 Thresholds before data]
|
v
[6.3 Practical activity — Lab K]
|
v
[6.4 Interoperability matrices]
|
v
[6.5 Validity]
|
v
[6.6 Worked case]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX026]
Learning outcomes
You will (1) design controlled evaluations of migration changes; (2) set decision thresholds before collecting data; (3) use confidence bounds for staged rollout decisions; (4) evaluate interoperability across a heterogeneous estate; and (5) report evaluation results with their validity limits.
6.1 Evaluation questions
Specialist evaluation answers decision questions: does enabling a hybrid group on the PACS gateway keep failure rates within budget? Does ML-DSA in the internal PKI break any clinical application? Which VPN firmware versions negotiate the intended methods? Each question needs a population, a measure, a threshold set in advance and a decision rule.
At NRHT the PACS gateway question was answered with a population of real client types, a failure-rate measure, a threshold of 0.09 % agreed in advance and a rule to promote, hold or exclude. Write the same four elements for each question before collecting data, and file them with the change record.
6.2 Thresholds before data
Set failure budgets and latency thresholds before the experiment, agreed with service owners. Thresholds set afterwards drift toward whatever the data show. A typical budget: handshake failure rate may rise by at most 0.04 percentage points over baseline, judged by the 95 % upper confidence bound.
Agree the budget with the service owner in writing and record it before the first canary. At NRHT the budget of 0.04 percentage points over a 0.05 % baseline produced the 0.09 % threshold used for the PACS gateway. If the budget needs changing, change it before the next stage, not after looking at results.
6.3 Practical activity — Lab K
# Lab K (ch 6): canary gate for enabling a hybrid group - promote only if failures stay within budget.
import math
def upper95(k, n): # Wilson upper bound
z = 1.96; p = k / n; d = 1 + z*z/n
return ((p + z*z/(2*n)) + z*math.sqrt(p*(1-p)/n + z*z/(4*n*n))) / d
baseline, budget = 5e-4, 4e-4 # baseline failure rate, allowed increase
for stage, k, n in [("1% canary", 3, 5000), ("10% canary", 28, 50000), ("10% canary (proxy site)", 96, 50000)]:
ub = upper95(k, n); ok = ub <= baseline + budget
print(f"{stage:24s} {k}/{n} upper95={ub:.5f} {'PROMOTE' if ok else 'HOLD'}")
Expected output:
1% canary 3/5000 upper95=0.00176 HOLD
10% canary 28/50000 upper95=0.00081 PROMOTE
10% canary (proxy site) 96/50000 upper95=0.00234 HOLD
The first stage holds on uncertainty, not on evidence of harm; the proxy site holds on evidence of harm. These are different findings and deserve different actions — extend the first stage; investigate the second.
The gate uses the Wilson upper bound because small failure counts make simpler intervals unreliable. The first stage holds only because 5,000 attempts are too few to show the rate is within budget. The proxy site holds because its upper bound is well above budget. Record both decisions with the stage size and the date.
6.4 Interoperability matrices
Hospitals have long tails of clients: old browsers on clinical workstations, embedded browsers in devices, partner systems. An interoperability matrix lists client types × server configurations with observed outcomes. Build it in a lab with representative clients first, then confirm with production telemetry. Gaps in the matrix are unknowns to resolve before broad rollout.
At NRHT the lab matrix of 14 client types found two embedded viewers that failed with the larger ClientHello before any production rollout. Keep the matrix with dates and versions, and update it as clients are upgraded or retired. A cell marked unknown blocks broad rollout until it is filled.
6.5 Validity
Lab evaluations have external-validity limits: real inspection proxies, Wi-Fi conditions and device firmware differ. Production canaries have internal-validity limits: concurrent changes and weekly cycles. Report both and prefer conclusions supported by lab and canary evidence together.
State in each evaluation report which conclusions rest on lab evidence, which on canary evidence and which on both. At NRHT the inspection proxy problem appeared in a production canary, not in the lab, which is why both kinds of evidence are needed. List concurrent changes during the canary period.
6.6 Worked case
NRHT evaluates hybrid key exchange on the PACS gateway. Pre-set threshold: upper bound ≤ 0.09 % failures. Lab matrix: 14 client types; two embedded viewers fail with the larger ClientHello. Canary 1 %, extended to two weeks until the upper bound fell within the threshold: promote. Canary 10 %: one hospital exceeds the threshold — its legacy inspection proxy. Decision: exclude that hospital, upgrade the proxy, continue elsewhere; the two embedded viewers are routed through a classical-only listener with documented residual risk and a vendor ticket.
The decision treats each problem separately, which pooled results would have hidden. Record the excluded hospital and the classical-only listener in the exception register, each with an owner and an expiry date.
6.7 Common errors and limits
- Setting thresholds after seeing data.
- Pooling sites, which hides a failing one.
- Lab-only conclusions for production decisions.
Each error has a guard in the PACS evaluation: a threshold agreed before any data, results reported by site, and decisions based on both the lab matrix and production canaries. A pooled result that looks acceptable can hide a site such as the hospital with the legacy proxy. Write the threshold, the site breakdown and the evidence source into every rollout decision, so a reviewer can check each one.
Exercises
- EX026 — Decision rule. Write a decision rule for promoting a canary that includes threshold, statistic and minimum duration. Rubric: 3 = all three, justified.
- EX027 — Interpret holds. Explain the different actions for Lab K's first and third holds. Rubric: 3 = extend sample vs investigate cause.
- EX028 — Matrix design. List six client types NRHT must include in its interoperability matrix. Rubric: 3 = six realistic, distinct types.
- EX029 — Pooling. Give a numeric example where pooled results pass but one site fails. Rubric: 3 = correct arithmetic showing masking.
- EX030 — Validity statement. Write a validity paragraph for the PACS evaluation. Rubric: 3 = internal and external limits with effects.
Synthesis
Controlled evaluation fixes thresholds in advance, uses confidence bounds, maps interoperability across real client populations and reports validity limits, so rollout decisions rest on evidence rather than hope.
Continue studying — chapter 06 of 12
← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →