Chapter 29 — Quantum and post-quantum security: operations
Lesson map
Lesson 29 published. Map, example, EX057.
flowchart TD
N0["Learning outcomes"]
N1["29.1 Staged rollout"]
N2["29.2 Practical activity — Lab K"]
N3["29.3 Monitoring"]
N4["29.4 Communication"]
N5["29.7 Rollback in practice"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX057"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["29.1 Staged rollout"]
N2["29.2 Practical activity — Lab K"]
N3["29.3 Monitoring"]
N4["29.4 Communication"]
N5["29.7 Rollback in practice"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX057"]
[Learning outcomes]
|
v
[29.1 Staged rollout]
|
v
[29.2 Practical activity — Lab K]
|
v
[29.3 Monitoring]
|
v
[29.4 Communication]
|
v
[29.7 Rollback in practice]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX057]
Learning outcomes
You will (1) plan a staged rollout with canaries; (2) use confidence bounds for promotion decisions; (3) monitor negotiated algorithms in production; and (4) coordinate change with help desk and partners.
29.1 Staged rollout
Operations introduce change gradually: enable a hybrid group for 1 % of traffic, then 10 %, then everyone, checking at each stage that failure rates stay within an agreed budget. Each stage has a duration long enough to see weekly patterns and a named decision-maker.
The budget and the stage durations are agreed before the first stage starts, so that pressure to finish does not change them midway. HCU sets its budget against the baseline failure rate measured on the gateway before any change, and it treats a site with unusual equipment as its own stage rather than averaging it into the rest.
29.2 Practical activity — Lab K
# Lab K (ch 29): canary gate for enabling a hybrid group - promote only if failures stay within budget.
import math
def upper95(k, n): # Wilson upper bound
z = 1.96; p = k / n; d = 1 + z*z/n
return ((p + z*z/(2*n)) + z*math.sqrt(p*(1-p)/n + z*z/(4*n*n))) / d
baseline, budget = 5e-4, 4e-4 # baseline failure rate, allowed increase
for stage, k, n in [("1% canary", 3, 5000), ("10% canary", 28, 50000), ("10% canary (proxy site)", 96, 50000)]:
ub = upper95(k, n); ok = ub <= baseline + budget
print(f"{stage:24s} {k}/{n} upper95={ub:.5f} {'PROMOTE' if ok else 'HOLD'}")
Expected output:
1% canary 3/5000 upper95=0.00176 HOLD
10% canary 28/50000 upper95=0.00081 PROMOTE
10% canary (proxy site) 96/50000 upper95=0.00234 HOLD
The decision uses the upper confidence bound, not the point estimate. The first canary's point estimate is low but the sample is small; the bound exceeds the budget, so the team waits for more data. The proxy-site canary shows a real problem.
The second canary has a larger sample, so its bound is tight enough to promote. The third has the same sample size but more failures, and its bound is well above the budget, so the team holds and investigates the site instead of waiting for more data.
29.3 Monitoring
Production monitoring should report negotiated key-exchange groups by service and site, fallback shares, handshake failures by error type, and certificate validation errors. Alerts fire when fallback exceeds thresholds. Measurement lag matters: if partner data arrives monthly, promotion decisions for partner flows cannot be weekly.
Dashboards should separate sites, so that one problem location does not disappear in an average. HCU's monitoring reports fallback share for each branch, which is how the proxy problem in its rollout was found quickly. Fallback that grows slowly is a warning sign too.
29.4 Communication
Publish a short note at each stage: what changed, measured failure bound, known exceptions and next decision date. Help-desk staff who know that a hybrid rollout reached a site will route related tickets quickly.
Notes are short and go to the same list each time, so that staff know where to look. HCU sends them to branch managers and the help desk, and keeps them with the rollout record as part of the evidence of what changed and when. When a stage is held, the note says so plainly and gives the reason, so nobody assumes the rollout has quietly moved on.
29.5 Worked example
HCU rolls out hybrid key exchange on its web gateway. Stage 1 (1 %, one week) holds because the confidence bound is wide; stage 1 extended to two weeks promotes. Stage 2 at 10 % reveals elevated failures from one branch with a legacy inspection proxy; that branch is excluded, the proxy vendor is engaged and the rest proceeds. Monitoring shows 0.3 % fallback after full rollout, all from old clients.
The excluded branch has its own exit date, after which it is reviewed again. The note for each stage went to the help desk the same day.
29.6 Common errors and limits
- Promoting on point estimates from small samples.
- Monitoring only failures, not fallback.
- Changing configuration without help-desk awareness.
- Averaging all sites together, so one failing location is hidden.
- Holding a stage without recording a reason or a new decision date.
- Leaving an excluded site without an owner or a deadline, so it never rejoins the rollout.
- Setting the failure budget after seeing the data rather than before.
Each of these turns a measured, staged change into a rollout that nobody can fully explain later.
29.7 Rollback in practice
Every stage needs a rehearsed rollback: the exact configuration to restore, who may trigger it, and how long it takes. Rehearse it in the lab before the first canary. A rollback that has never been tested is a hope, not a control. After a rollback, record what was observed, keep the evidence, and decide whether the cause was the algorithm, the implementation or the environment before trying again; most first-attempt rollbacks in hybrid deployments trace to environment issues such as old proxies.
Exercises
- EX057 — Promotion rule. Explain why Lab K's first canary is held even though 3/5,000 = 0.06 % is below the 0.09 % threshold. Rubric: 3 = upper bound exceeds budget due to small sample.
- EX058 — Monitoring set. List four metrics for monitoring a hybrid rollout and the alert condition for one. Rubric: 3 = four relevant metrics, one sensible alert.
Synthesis
Staged rollouts with confidence-bound gates, monitoring of negotiated algorithms and clear communication turn a risky change into a controlled one.
Continue studying — chapter 29 of 40
← Chapter 28 (draft) · Course chapter index · Chapter 30 (draft) →