Chapter 37 — Quantum and post-quantum security: operations
Lesson map
Lesson 37 published. Map, example, EX073.
flowchart TD
N0["Learning outcomes"]
N1["37.1 Canary gates"]
N2["37.2 Practical activity — Lab K"]
N3["37.3 Monitoring"]
N4["37.4 Certificate and key operations"]
N5["Engineering note"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX073"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["37.1 Canary gates"]
N2["37.2 Practical activity — Lab K"]
N3["37.3 Monitoring"]
N4["37.4 Certificate and key operations"]
N5["Engineering note"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX073"]
[Learning outcomes]
|
v
[37.1 Canary gates]
|
v
[37.2 Practical activity — Lab K]
|
v
[37.3 Monitoring]
|
v
[37.4 Certificate and key operations]
|
v
[Engineering note]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX073]
Learning outcomes
You will (1) operate staged rollouts with statistical gates; (2) monitor negotiated algorithms and fallback; (3) manage certificates and keys for new algorithms; and (4) run change and incident processes. It builds on Chapters 17, 32 and 36.
37.1 Canary gates
Promote a rollout stage only when the upper confidence bound of the failure rate is within the budget. Small canaries cannot demonstrate small effects — the bound is wide — so the gate holds until enough traffic has been observed.
The gate compares the upper bound with the baseline plus the allowed increase, not with zero, because some handshakes fail for ordinary reasons such as clients dropping off mobile networks. Measuring the baseline for a week before the change is therefore part of the gate. Without it, a team cannot tell whether a failure rate of one in two thousand is new or has always been there.
37.2 Practical activity — Lab K
# Lab K (ch 37): canary gate for enabling a hybrid group - promote only if failures stay within budget.
import math
def upper95(k, n): # Wilson upper bound
z = 1.96; p = k / n; d = 1 + z*z/n
return ((p + z*z/(2*n)) + z*math.sqrt(p*(1-p)/n + z*z/(4*n*n))) / d
baseline, budget = 5e-4, 4e-4 # baseline failure rate, allowed increase
for stage, k, n in [("1% canary", 3, 5000), ("10% canary", 28, 50000), ("10% canary (proxy site)", 96, 50000)]:
ub = upper95(k, n); ok = ub <= baseline + budget
print(f"{stage:24s} {k}/{n} upper95={ub:.5f} {'PROMOTE' if ok else 'HOLD'}")
Expected output:
1% canary 3/5000 upper95=0.00176 HOLD
10% canary 28/50000 upper95=0.00081 PROMOTE
10% canary (proxy site) 96/50000 upper95=0.00234 HOLD
The 1% canary holds not because failures are high but because 5,000 connections cannot bound the rate tightly. The proxy site holds because its rate is genuinely higher — a local problem to fix, not a reason to roll back globally.
The gate passes when the bound is at most 0.0009, the baseline of 0.0005 plus the budget of 0.0004. At 3 failures in 5,000 the point estimate is 0.0006, already within budget, but the Wilson bound of 0.00176 is not. Try 0 and 1 failures in 5,000: zero gives a bound of about 0.00077 and promotes, while a single failure gives 0.00113 and holds. At the baseline rate alone, 5,000 connections would see about 2.5 failures, so a canary this small almost always holds and the gate simply waits for more traffic.
37.3 Monitoring
Dashboards: negotiated groups by service, fallback rate and reasons, handshake failures by path, certificate algorithms in use, stateful key capacity, library versions. Alerts on fallback spikes and on any unexpected classical-only negotiation where hybrids are required.
Each dashboard item answers an operational question: is the hybrid group being negotiated, where are failures concentrated, which libraries need updates, and how many signatures remain on stateful keys. Alerts focus on changes, so the on-call engineer looks at what moved since yesterday instead of reading every panel. BPA reviews whether each alert led to useful action and removes alerts that never do.
37.4 Certificate and key operations
Certificate lifecycle automation (for example ACME, RFC 8555) makes algorithm changes routine: short-lived certificates can switch algorithms at renewal. Key ceremonies for new PQC roots are planned like any root ceremony, with witnesses and recorded procedures.
With ninety-day certificates issued through ACME, BPA can move a service to a new signature algorithm within one renewal cycle once clients support it. A root ceremony takes far longer to arrange, so the post-quantum root ceremony is scheduled early, with the HSM and witnesses booked months ahead.
37.5 Worked example
BPA's operations runbook for hybrid TLS: daily dashboard review during rollout; promote via gates; on gate hold, investigate by path; on fallback spike, check recent changes; on proxy-site failures, apply path-specific exception with expiry and open vendor ticket.
Each step of the runbook maps to a lesson from the case: gates come from Lab K, investigation by path from the proxy-site result, and path-specific exceptions from the shipping-line portal fault. Writing these steps down means the night shift handles a gate hold the same way the rollout team would. The runbook is reviewed after each incident and updated when a step proved unclear.
37.6 Common errors and limits
- Promoting on point estimates from small canaries.
- Global rollback for local faults.
- Manual certificate processes that make algorithm change slow.
A small canary still has a wide confidence bound, as Lab K showed for 5,000 connections, where 3 failures gave a point estimate of 0.0006, inside the budget, but an upper bound of 0.00176, so promoting on the point estimate is guesswork. A global rollback for the proxy site's problem would remove protection from every healthy path. Manual certificate processes make each algorithm change a project, while automated renewal turns it into a routine step that happens every ninety days.
Engineering note
Operations also own communication during rollout. Publish a short status note each promotion stage listing the stage, measured failure bound, known exceptions and next decision date, so that help-desk staff can recognize related tickets and partners can plan. A help desk that knows a hybrid rollout reached the shipping-line portal's inspection proxy yesterday will route a cluster of connection complaints correctly in minutes instead of treating them as unrelated faults for days.
Exercises
- EX073 — Gate arithmetic. With baseline 5e-4 and budget 4e-4, is 30 failures in 50,000 connections promotable? Use Lab K's function. Rubric: 3 = correct decision with upper bound shown.
- EX074 — Runbook. Write the runbook section for "fallback rate doubled overnight". Rubric: 3 = triage steps, owners, escalation.
Synthesis
Operations turns rollout into controlled steps with statistical gates, monitoring of negotiated outcomes, automated certificate lifecycles and path-specific responses.
Continue studying — chapter 37 of 50
← Chapter 36 (draft) · Course chapter index · Chapter 38 (draft) →