Chapter 27 — Deployment and validation
Lesson map
Lesson 27 published. Map, example, EX079.
flowchart TD
N0["Learning outcomes and connection"]
N1["27.1 Pre-checks"]
N2["27.2 Canaries and promotion criteria"]
N3["27.3 Validation after deployment"]
N4["27.4 What the window cannot show"]
N5["27.6 Practical activity: canary decision"]
N6["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX079"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["27.1 Pre-checks"]
N2["27.2 Canaries and promotion criteria"]
N3["27.3 Validation after deployment"]
N4["27.4 What the window cannot show"]
N5["27.6 Practical activity: canary decision"]
N6["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX079"]
[Learning outcomes and connection]
|
v
[27.1 Pre-checks]
|
v
[27.2 Canaries and promotion criteria]
|
v
[27.3 Validation after deployment]
|
v
[27.4 What the window cannot show]
|
v
[27.6 Practical activity: canary decision]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX079]
Learning outcomes and connection
After this chapter you should be able to:
- prepare a deployment with pre-checks, including verification of what is being deployed;
- deploy in stages with a canary and explicit promotion and rollback criteria;
- validate after deployment against the service's functional and security properties;
- state what a validation window can and cannot show.
Chapter 14 planned changes; Chapter 22 automated configuration. This chapter is about software deployments — a new relay version, a historian upgrade — where the change is larger and harder to predict. In critical services, the goal is to find a problem on one host, at a quiet time, with a rollback ready, rather than on every host at the morning peak.
27.1 Pre-checks
Before deploying: the change is approved; operations know the window; a configuration backup has been taken and the rollback package is staged; and the package itself is verified — its signature or published digest checked against the supplier's, so that what you install is what the supplier released. Secure-development guidance asks producers to protect the integrity of released software, for example with signatures (NIST SP 800-218); as an implementer, your side of that is to check them, and to treat an unverifiable package as a finding.
27.2 Canaries and promotion criteria
A canary deployment installs the new version on one host first and compares it with hosts still on the old version, over the same period and load. Google's SRE workbook describes canarying as a partial, time-limited deployment evaluated against a control population before wider rollout (Google SRE workbook, Canarying releases). Write promotion criteria before deploying: for LANTERN, acceptance ratio not more than 0.2 percentage points below control; p95 latency not more than 25 % above control; no restarts; MAC rejections unchanged. If any criterion fails, roll back the canary and open a problem record. If all pass, promote one host at a time.
27.3 Validation after deployment
After each host is upgraded, validate: functional properties (all stations accepted, latency normal, historian receiving); security properties (MAC checks still enforced — inject a test message with a wrong MAC; listeners unchanged — run the drift check; logs still forwarded and parsed — check the normalizer's counts); and operational properties (alerts still evaluated; health report green). Security regressions are easy to miss because a service can work perfectly while a check has been silently disabled.
On RLY-03, the drift check after 2.4.1 still showed the debug listener: function passed, a security property did not.
27.4 What the window cannot show
A one-hour night-time canary shows behaviour at night-time load for one hour. It says nothing about the morning demand peak, weekly report runs (Chapter 1), month-end reporting or rare message types. Either extend the window over the periods that matter or state the gap and watch those periods closely after promotion.
Relayd 2.4.1's canary ran one night-time hour, so its 0.0010 drop in acceptance ratio was recorded as an open item for the morning peak.
Weekly report runs on days 6 and 7, at about 62 % CPU, are another period an hour at night cannot show.
27.5 Worked example: relayd 2.4.1 (synthetic)
Pre-checks pass: approved change, control room informed, configuration backed up, rollback package staged, package signature verified. The canary RLY-03 runs 2.4.1 for an hour against RLY-01 on 2.4.0. Acceptance ratio 0.9985 versus 0.9994 (a drop of 0.0010, within 0.002); p95 latency 41 ms versus 37 ms (within 25 %); no restarts; MAC rejects unchanged. Decision: promote to RLY-02, then RLY-01. Post-deployment validation on RLY-03 injects a wrong-MAC test message (rejected, as expected) and runs the drift check — which still shows Chapter 22's debug listener, a reminder that the deployment did not fix configuration drift and should not be reported as if it did. The acceptance-ratio drop, though within criteria, is recorded as an open item to check at the morning peak.
27.6 Practical activity: canary decision
Lab 27 (save as lab27.py, run with python3 lab27.py):
# Lab 27 - pre- and post-deployment validation with a canary and a rollback rule (offline, standard library only)
# Synthetic: relayd 2.4.1 rolled to rly-03 first (canary), compared with rly-01 still on 2.4.0 for one hour.
checks_pre = {"backup of config taken": True, "rollback package staged": True, "change ticket approved": True,
"operations told (control room)": True, "maintenance window open": True, "signature of package verified": True}
canary = {"msgs_in": 7200, "msgs_accepted": 7189, "mac_rejects": 0, "p95_latency_ms": 41, "cpu_p95": 38, "restarts": 0}
control = {"msgs_in": 7200, "msgs_accepted": 7196, "mac_rejects": 0, "p95_latency_ms": 37, "cpu_p95": 36, "restarts": 0}
limits = {"accept_ratio_drop": 0.002, "latency_ratio": 1.25, "restarts": 0}
print("pre-checks:", "PASS" if all(checks_pre.values()) else "FAIL " + str([k for k, v in checks_pre.items() if not v]))
ra, rc = canary["msgs_accepted"] / canary["msgs_in"], control["msgs_accepted"] / control["msgs_in"]
results = {
"acceptance ratio": (rc - ra <= limits["accept_ratio_drop"], f"{ra:.4f} vs {rc:.4f} (drop {rc-ra:.4f})"),
"p95 latency": (canary["p95_latency_ms"] <= control["p95_latency_ms"] * limits["latency_ratio"], f"{canary['p95_latency_ms']} vs {control['p95_latency_ms']} ms"),
"restarts": (canary["restarts"] <= limits["restarts"], str(canary["restarts"])),
"MAC rejects unchanged": (canary["mac_rejects"] == control["mac_rejects"], str(canary["mac_rejects"])),
}
for k, (ok, detail) in results.items(): print(f" {k:<22} {'pass' if ok else 'FAIL'} {detail}")
print("decision:", "promote to rly-02, then rly-01" if all(ok for ok, _ in results.values()) else "ROLL BACK canary and open problem record")
# What the canary cannot show: a 1-hour window at night says nothing about the morning demand peak or month-end reports.
Expected output (Python 3.13.5):
pre-checks: PASS
acceptance ratio pass 0.9985 vs 0.9994 (drop 0.0010)
p95 latency pass 41 vs 37 ms
restarts pass 0
MAC rejects unchanged pass 0
decision: promote to rly-02, then rly-01
Check each line against the criteria fixed before the window: acceptance ratio 0.9985 against 0.9994, a drop of 0.0010 within the 0.002 allowed; p95 latency 41 ms against 37 ms, within 25 %; no restarts; MAC rejects unchanged. All pass, so the decision is to promote to RLY-02, then RLY-01, one host at a time. The drop, though within the criteria, stays on the open-items list for the morning peak.
If the drop had been 0.0025, the rule fixed in advance would have required rollback, however small the difference looked.
27.7 Common errors and limits
- Installing packages without verifying their origin.
- Promotion criteria decided after seeing canary results.
- Validating function but not security properties.
- A canary window that misses the periods of real load.
- Promoting all remaining hosts at once after the canary.
- Rollback never tested.
The relayd 2.4.1 deployment avoided each: signature verified, criteria fixed before the canary, a wrong-MAC message injected after, and promotion one host at a time.
The open item for the morning peak is how the fourth is handled when the window cannot be longer.
Exercises — chapter 27
EX079 — Fail the canary. Objective: apply the rollback rule. Inputs: Lab 27. Constraints: change only the canary's numbers. Steps: (1) set the canary's accepted count to 7170; (2) run; (3) state the decision and the first three actions. Expected evidence: output and actions. Rubric: 3 = acceptance drop ≈ 0.0036 fails the 0.002 criterion, decision roll back; actions: roll back canary under the change, preserve canary logs and reject reasons, open problem record and inform control room; 2 = decision right, actions thin; 1 = not run; 0 = none.
EX080 — Security validation set. Objective: specify post-deployment security checks. Inputs: Chapter 27 ("Validation after deployment"). Constraints: five checks, each with expected result. Steps: list them. Expected evidence: table. Rubric: 3 = wrong-MAC test rejected, replay test rejected, drift check shows no new listeners, normalizer unparsed count unchanged, service runs as unprivileged account, alert test still fires; 2 = four; 1 = function only; 0 = none.
EX081 — Extend the window. Objective: design a canary window that covers real load. Inputs: weekly report runs on days 6–7 (Chapter 1), morning peak 06:00–09:00. Constraints: the change must complete within ten days. Steps: propose the window(s), promotion points and the criteria checked at each. Expected evidence: schedule. Rubric: 3 = canary across at least one morning peak and one report run before promotion, with criteria evaluated at each, then staged promotion within the ten days; notes rollback readiness throughout; 2 = covers one; 1 = one-hour window kept; 0 = none.
Chapter synthesis
Deploy in stages. Pre-check approvals, operations, backups, rollback and the package's origin. Canary on one host against a control with promotion criteria fixed in advance; roll back on any failure and promote one host at a time on success. Validate functional, security and operational properties after each host, and state honestly what the validation window could not show.
Continue studying — chapter 27 of 30
← Chapter 26 · Course chapter index · Chapter 28 →
Continue to the next authored lesson.