Chapter 27 — Deployment and validation
Lesson map
Lesson 27 published. Map, example, EX053.
flowchart TD
N0["Learning outcomes"]
N1["27.1 Stages and gates"]
N2["27.2 Validation in production"]
N3["27.3 Exceptions"]
N4["27.4 Closing"]
N5["27.6 Practical activity"]
N6["27.7 Change freeze and timing"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX053"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["27.1 Stages and gates"]
N2["27.2 Validation in production"]
N3["27.3 Exceptions"]
N4["27.4 Closing"]
N5["27.6 Practical activity"]
N6["27.7 Change freeze and timing"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX053"]
[Learning outcomes]
|
v
[27.1 Stages and gates]
|
v
[27.2 Validation in production]
|
v
[27.3 Exceptions]
|
v
[27.4 Closing]
|
v
[27.6 Practical activity]
|
v
[27.7 Change freeze and timing]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX053]
Learning outcomes
You will (1) plan production deployment with stages, gates and rollback; (2) validate end-to-end behaviour in production; (3) handle exceptions during rollout; and (4) close a deployment with evidence.
27.1 Stages and gates
Stage 1: canary listener or 1 % of traffic; stage 2: 10 %; stage 3: all tier-1 listeners; stage 4: remaining listeners. Gates: failure-rate upper bound within budget, fallback within expectation, no unexplained alert, runbooks ready. Each stage has a minimum duration covering weekly patterns.
KLS's stages last at least seven days so weekly batch jobs and partner uploads are included. Each gate is checked by someone outside the deploying team and recorded with its numbers, for example the Wilson upper bound and the fallback share at the end of the stage.
27.2 Validation in production
Validate with synthetic clients in each region and with production negotiation logs; confirm every segment of tier-1 flows. Sample certificates and configurations independently of the deploying team.
KLS runs synthetic clients from five regions every 15 minutes, each recording the group negotiated at the CDN edge, while origin logs show the group on the next hop. An engineer from another team samples ten listeners per stage, comparing the live profile hash and certificate algorithm with the inventory. Disagreements stop the stage until explained.
27.3 Exceptions
When a client population cannot cope — an old proxy, a partner client — create an exception: scope, reason, compensating control, owner, expiry. Route the population to a classical-compatible listener if needed, rather than weakening the default.
A dedicated listener keeps the default profile intact for everyone else: the exception population is routed by address range to a listener that still accepts X25519 only, with its own monitoring. The exception's expiry is enforced by the CI gate's exception file, so the listener's configuration fails the gate when the date passes.
27.4 Closing
Close the deployment with an evidence pack: stages and gate results, validation results, exceptions, residual risk, and the handoff acceptance.
KLS's closure pack is a folder in the change record with the stage table, each gate's numbers, the regional validation results, the exception list, the residual-risk entries and the signed handoff acceptance from SRE. The deployment is closed only when every item links to evidence. Anything open moves to a named follow-up ticket instead of keeping the change open indefinitely.
27.5 Worked example
KLS deploys kls-modern to all internet-facing listeners. Stage 2 shows one customer's corporate proxy failing; exception created with a dedicated listener for that customer's IP range, expiry six months, customer notified. Stage 3 passes; validation from five regions shows hybrid at edge and origin. Closure pack filed; residual risk: certificates classical.
The exception kept the customer working without weakening anyone else's profile, and the customer received the test endpoint so its proxy upgrade could be verified before the expiry. The closure pack recorded the remaining classical certificates as the next programme item.
27.6 Practical activity
Write a deployment plan with stages, gates, rollback and exception process for your studio flow.
Write the stages with traffic share and minimum duration; the gates for each with thresholds and the data source; the rollback for each stage and how long it takes; and the exception process with the fields an exception record needs. Add who checks each gate and how the result is recorded. State the scheduling constraint for your flow, such as avoiding a partner's month-end batch for SFTP. A reviewer should be able to run your plan without asking you a question.
27.7 Change freeze and timing
Avoid deploying cryptographic changes just before peak business periods — for a logistics platform, end-of-quarter shipping peaks — or during other major changes, because overlapping changes make failures hard to attribute. Schedule stages when the owning team is staffed to watch dashboards, and record the planned and actual timing in the closure pack.
KLS freezes cryptographic changes in the last two weeks of each quarter and avoids weeks with platform upgrades.
For KLS this also means no SFTP changes in the first days of a month, when partners run their largest uploads. Planned and actual times are compared in the closure pack, and any stage that slipped explains why.
27.8 Common errors and limits
- Weakening the default to accommodate one client.
- Stages shorter than weekly cycles.
- Deployments closed without validation evidence.
During KLS's rollout an engineer suggested dropping the hybrid group from the default profile to fix the one customer's proxy; the dedicated listener solved it without exposing every other client. A stage planned for three days would have missed the weekly partner batch that later produced the only SFTP failures. And an early closure lacked regional validation, so a CDN region still classical to origin went unnoticed until the synthetic clients were added.
Exercises
- EX053 — Gate list. Write four gates for stage 3. Rubric: 3 = measurable gates with thresholds.
- EX054 — Exception. Write the exception record for the customer proxy in this chapter's worked example. Rubric: 3 = scope, reason, compensating control, owner, expiry.
Synthesis
Staged deployment with gates, production validation, contained exceptions and an evidence-based closure delivers post-quantum protection without weakening defaults.
Continue studying — chapter 27 of 30
← Chapter 26 (draft) · Course chapter index · Chapter 28 (draft) →