◇ OpenCyber Academy
Technologist · Chapter 27 · English

Chapter 27 — Deployment and validation

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 27 — Deployment and validation

Lesson map

Lesson 27 published. Map, example, EX079.

flowchart TD
  N0["Learning outcomes"]
  N1["27.1 Staged rollout"]
  N2["27.2 Validating in production"]
  N3["27.3 Shadow mode"]
  N4["27.4 Go or no-go"]
  N5["27.7 Recording the go or no-go decision"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX079"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["27.1 Staged rollout"]
  N2["27.2 Validating in production"]
  N3["27.3 Shadow mode"]
  N4["27.4 Go or no-go"]
  N5["27.7 Recording the go or no-go decision"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX079"]

[Learning outcomes]
    |
    v
[27.1 Staged rollout]
    |
    v
[27.2 Validating in production]
    |
    v
[27.3 Shadow mode]
    |
    v
[27.4 Go or no-go]
    |
    v
[27.7 Recording the go or no-go decision]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX079]

Learning outcomes

You will (1) deploy AI changes safely with staged rollouts; (2) validate security controls in production without harming users; (3) use canaries and shadow modes; and (4) decide go or no-go.

27.1 Staged rollout

Deploy to a small percentage of traffic first, then increase while watching security and service metrics. Use feature flags so a tool or model can be switched off instantly. Keep the previous version ready to restore, and make sure the restore has been tested. Ask Brightwell's new prompt went first to 5% of traffic with hourly test-account probes before reaching full traffic. Feature flags let operations switch off the returns tool instantly. The previous prompt stayed ready to restore, and the restore was tested before the 5% stage began.

27.2 Validating in production

Security validation in production must not harm real users. Use synthetic test accounts and test orders; plant canaries only in documents that real customers cannot retrieve harmfully; run attack cases only against test accounts. Coordinate with operations and get authorisation for each test window. For Ask Brightwell, Jonas's production probes use test accounts and test orders that real customers cannot see. Canary documents sit in sources no customer query can reach, and each test window is agreed with operations in advance so that an alert during the window is recognised as part of the test.

27.3 Shadow mode

For classifiers like ReturnGuard, a new model can run in shadow: it scores real requests but its output is only logged, not used. Compare decisions with the current model, especially on known fraud and on groups that might be treated unfairly, before switching. ReturnGuard v2's shadow run showed why the group check matters: it flagged returns from one rural region far more often than v1 because of courier delays. The comparison used known fraud cases and region as the group of interest, both chosen before shadow mode began.

27.4 Go or no-go

Decide with criteria set before deployment: gate results passed; shadow comparison acceptable; monitoring live; runbooks rehearsed; owner sign-off. Missing any criterion means no-go or a smaller rollout, not a quiet exception. ReturnGuard v2 was a no-go after shadow mode because one criterion, an acceptable comparison across groups, was not met. The team held the rollout and added a delivery-delay feature, instead of releasing with a quiet exception. The decision and its reason were recorded for the next shadow run. Waiving the criterion was not considered, because it was set before shadow mode.

27.5 Worked example (synthetic Brightwell case)

ReturnGuard v2 runs in shadow for two weeks. It catches 18% more known fraud but flags returns from one rural region three times as often as v1 — courier delays there produce late-return patterns. The team holds the rollout, adds a delivery-delay feature, and reruns shadow mode. Ask Brightwell's new prompt goes to 5% of traffic with test-account injection probes every hour; after three days with zero policy denials above baseline, it reaches 100%. Both decisions followed pre-set criteria: ReturnGuard v2 failed the fairness comparison and was held; the prompt met its probe criteria and went ahead.

27.6 Errors and limits

Brightwell's releases avoid these. Ask Brightwell's prompt went to 5% first, all attack probes used test accounts, and ReturnGuard v2's no-go followed criteria set before shadow mode. A big-bang release would have exposed every customer to an untested prompt at once. ReturnGuard v2's hold shows that pre-set criteria work as intended: a model that caught more fraud was still stopped because it treated one region unfairly, and nobody could argue the threshold away after seeing the result.

27.7 Recording the go or no-go decision

The go or no-go meeting for a new Ask Brightwell version ends with a written decision: who decided, which criteria were met, which were waived and why, and the rollback trigger. That record is what an auditor or the next release team reads first. ReturnGuard v2's no-go record names the unmet group-comparison criterion and what must change before the next shadow run. The record also names who decided and the rollback trigger for the next attempt, so the next release team can see what changed between the two shadow runs and why the second may be approved.

Exercises

Synthesis

Deploy AI changes gradually behind flags, validate controls safely in production, use shadow modes for models and decide with pre-set criteria.

Continue studying — chapter 27 of 30

← Chapter 26 (draft) · Course chapter index · Chapter 28 (draft) →