Chapter 23 — Quantum and post-quantum security: evaluation
Lesson map
Lesson 23 published. Map, example, EX045.
flowchart TD
N0["Learning outcomes"]
N1["23.1 Objectives"]
N2["23.2 Performance"]
N3["23.3 Coverage and residual risk"]
N4["23.4 Report"]
N5["23.6 Practical activity"]
N6["23.7 Comparing with the baseline"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX045"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["23.1 Objectives"]
N2["23.2 Performance"]
N3["23.3 Coverage and residual risk"]
N4["23.4 Report"]
N5["23.6 Practical activity"]
N6["23.7 Comparing with the baseline"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX045"]
[Learning outcomes]
|
v
[23.1 Objectives]
|
v
[23.2 Performance]
|
v
[23.3 Coverage and residual risk]
|
v
[23.4 Report]
|
v
[23.6 Practical activity]
|
v
[23.7 Comparing with the baseline]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX045]
Learning outcomes
You will (1) evaluate a deployment against its objectives; (2) measure performance impact; (3) assess residual risk and coverage; and (4) write an evaluation report for operations and management.
23.1 Objectives
Typical objectives: tier-1 flows negotiate hybrid key exchange end to end; failure rates within budget; latency within budget; fallback below threshold; residual risks documented; monitoring and runbooks in place.
KLS writes each objective with a threshold and a source, for example "tier-1 flows hybrid on every segment, from per-hop logs".
Others: failure rate within the 0.04-point budget, 99th-percentile latency increase under 5 ms, fallback below 5 % excluding listed exceptions, and runbooks exercised once. Each threshold is agreed before deployment, so the evaluation cannot move the goalposts afterwards.
23.2 Performance
Measure handshake latency at the median and 99th percentile, CPU per handshake on servers and constrained clients, and bytes per handshake. ML-KEM operations are fast — typically comparable to or faster than elliptic-curve operations on servers — so the visible costs are usually bytes and occasional extra round trips, not CPU. Measure on representative paths, including mobile and warehouse networks.
KLS measured bytes per handshake directly: about 2.3 kB more with the hybrid, matching the 1,184-byte and 1,088-byte additions in each direction. On the warehouse network that pushed some hellos into a second segment; CPU on the gateway rose by less than the measurement noise.
23.3 Coverage and residual risk
Report the share of tier-1 flows fully migrated (all segments), partly migrated and unknown. List residual risks with owners. Compare with the start of the programme.
KLS counts a flow as fully migrated only when every segment has evidence of hybrid key exchange; one classical or unknown segment makes it partial or unknown. Six months in, that rule gave 9 full, 4 partial and 1 unknown of 14 tier-1 flows. The comparison with the start, when only the browser-to-CDN segment was hybrid, is what management needed to see progress honestly.
23.4 Report
An evaluation report states objectives, methods, results, deviations, residual risk and recommendations, with evidence links. Operations need detail; management needs a one-page summary.
KLS's report fits on two pages for operations and one for management. The management page leads with the coverage figures, the residual risks with owners and the next decision needed, the internal PKI pilot. Every number links to the query that produced it, so a reader can rerun it.
The warehouse scanners appear on the first page as a deviation: they stay classical until the four-year vendor refresh, which by the Mosca estimate makes them late for any arrival date below fourteen years.
23.5 Worked example
KLS evaluation (synthetic) after six months: tier-1 flows 9 of 14 fully migrated, 4 partly (provider or partner segments), 1 unknown (forklift telemetry). Failure rate +0.01 points (within budget). API handshake p50 +0.6 ms, p99 +2.8 ms; warehouse scanners not migrated. Residual: certificates classical everywhere; payment-provider calls classical. Recommendation: internal PKI pilot next; continue partner adoption tracking.
The result shows why median latency alone misleads: the median changed by 0.6 ms, while the 99th percentile changed by 2.8 ms, which is where mobile clients and HelloRetryRequest round trips show up. The failure rate stayed within the 0.04-point budget used in the canary gates.
23.6 Practical activity
Write a one-page evaluation summary for your lab deployment with objectives, results and residual risk.
Use a lab deployment from earlier chapters: the hybrid profile on a lab ingress with a client matrix. State three objectives with thresholds; report results with numbers and sources, including at least one median and one 99th percentile; list residual risks, such as classical certificates and any client that negotiated X25519; and end with one recommendation and the decision it needs. Keep it to one page. If a result has no baseline, say so and describe how you would capture one.
23.7 Comparing with the baseline
An evaluation is only meaningful against a baseline captured before the change: failure rates, latency distributions, CPU and bytes per handshake on each path. If no baseline exists, capture one now from a classical-only control group — for example, a small share of traffic or a staging replica — before drawing conclusions. Without a baseline, an apparent regression may be normal weekly variation, and an apparent improvement may be a quiet week.
KLS kept 5 % of portal traffic on a classical-only listener for two weeks as its control group, then retired it.
23.8 Common errors and limits
- Counting partly migrated flows as done.
- Measuring only median latency.
- Evaluations without a baseline.
KLS's draft report first counted the four partial flows as done because their KLS-owned hops were finished; the external hops were still classical, so the claim was wrong. Its first latency figure was a median only, which hid the tail cost on mobile paths. And the comparison with "before" used remembered numbers rather than a captured baseline, so the apparent zero change in failures could not be trusted until a classical control group was measured.
Exercises
- EX045 — Coverage figure. Express KLS's tier-1 status as percentages of the 14 flows. Rubric: 3 = 64 % full, 29 % partial, 7 % unknown.
- EX046 — Performance claim. Why are bytes and round trips usually more visible than CPU for ML-KEM? Rubric: 3 = fast operations vs larger messages and HRR/segments.
Synthesis
Evaluation compares results with objectives and a baseline, measures tail latency and bytes, and reports full, partial and unknown coverage with residual risk.
Continue studying — chapter 23 of 30
← Chapter 22 (draft) · Course chapter index · Chapter 24 (draft) →