Chapter 27 — Deployment and validation | Technologist Cloud and container security

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 27 — Deployment and validation

Lesson map

Lesson 27 published. Map, example, EX079.

flowchart TD
  N0["Learning outcomes"]
  N1["27.1 Handoff checklist"]
  N2["Lab — handoff check"]
  N3["27.2 Reading the result"]
  N4["27.3 Restore test"]
  N5["27.4 Client contact"]
  N6["27.5 Validation in production"]
  N7["Why a checklist"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX079"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["27.1 Handoff checklist"]
  N2["Lab — handoff check"]
  N3["27.2 Reading the result"]
  N4["27.3 Restore test"]
  N5["27.4 Client contact"]
  N6["27.5 Validation in production"]
  N7["Why a checklist"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX079"]

[Learning outcomes]
    |
    v
[27.1 Handoff checklist]
    |
    v
[Lab — handoff check]
    |
    v
[27.2 Reading the result]
    |
    v
[27.3 Restore test]
    |
    v
[27.4 Client contact]
    |
    v
[27.5 Validation in production]
    |
    v
[Why a checklist]
    |
    v
[Worked example] --> [Exercise EX079]

Learning outcomes

You will (1) check a handoff pack for completeness; (2) validate controls in production; (3) close gaps.

27.1 Handoff checklist

Kwame provides a list of eight required items for any handoff. The eight items are a runbook, an owner list, alert routes, the baseline file, the exception register, a restore test record, a client contact and the known limits. Each answers a question the new owners will face: how to run a check, who decides, where alerts go, what is expected, what is accepted, whether data can be recovered, whom to tell and what is not covered. Kwame's list is the same for every handoff at Cobalt, not written for Priya.

Lab — handoff check

"""Check a synthetic handoff pack for required items (stdlib only)."""
required = ["runbook", "owner list", "alert routes", "baseline file", "exception register",
            "restore test record", "contact for client", "known limits"]
pack = {"runbook", "owner list", "alert routes", "baseline file", "exception register", "known limits"}
missing = [r for r in required if r not in pack]
print(f"handoff items {len(required)-len(missing)}/{len(required)} present")
for m in missing: print("  missing:", m)

Run with python3 lab_handoff.py. Verified output:

handoff items 6/8 present
  missing: restore test record
  missing: contact for client

27.2 Reading the result

Six of eight items are present. Missing: the restore test record and the client contact. Both missing items link back to work earlier in the placement. The restore test record is missing because the control mapping found that no restore had ever been tested, and the task had not yet been done. The client contact is missing because the account manager had not named one for this pack, so a finding affecting Riverside Dental would have had no clear recipient.

27.3 Restore test

Priya and the platform team restore a synthetic copy of Riverside Dental's database in a test account and time it at 40 minutes. The restore uses a synthetic copy in a separate test account, so no client data or production service is touched. Timing it gives the client and Cobalt a realistic figure for recovery instead of an assumption. The forty minutes include the undocumented step of re-creating a database user, which is then written into the runbook so the next restore does not depend on remembering it.

27.4 Client contact

The account manager adds the named client contact. Naming a specific person at Riverside Dental, with a backup, means a finding that affects the client reaches someone able to approve a change window or answer questions. The contact is recorded in the pack and in the runbooks' contact sections, so both the on-call engineer and the account manager use the same name. The account manager confirms the details with the client before adding them. Kwame checks the entry during the second run of the handoff check.

27.5 Validation in production

Each check is run in production once with Kwame watching, results compared with staging. Running each check once in production with Kwame watching confirms that staging results carry over, and that the checks still only read data. Comparing outputs with staging shows differences that staging alone would hide, such as the newly added namespace that the inventory script missed. Each production validation is recorded with its date, result and observer. Kwame signs each record, so the production results carry an independent witness.

Why a checklist

Handoffs at Cobalt had previously missed items, such as restore tests. A fixed checklist makes gaps visible and avoids relying on memory. The lab automates the check, but the important part is the list itself, agreed by the team.

After the gaps are closed, the pack is re-checked and passes 8 of 8.

Restore lessons

The restore test showed one undocumented step: re-creating a database user. It is now in the runbook.

Validation records

Each production validation is recorded with date, result and who observed it.

Second look

After closing the gaps, a colleague who had not seen the pack reviews it and confirms it is understandable without Priya. The colleague suggests adding a diagram of where each check runs, which Priya adds.

Sign-off

Kwame signs off the pack.

Final state

With all eight items present and production validated, the checks are formally owned by the platform team.

Worked example

Production validation shows the inventory script misses a workload in a newly added namespace. Priya changes it to read all namespaces, re-runs and confirms the count.

Synthetic case. Pack is synthetic.

Errors and limits

Staging does not always match production; validate in both.

Exercises

EX079 — Lab gaps. Which handoff items were missing? Rubric: 3 = restore test record, client contact; 2 = one; 1 = vague; 0 = none.
EX080 — Restore time. How long did the restore take? Rubric: 3 = 40 minutes; 2 = partial; 1 = vague; 0 = none.
EX081 — Production finding. What did production validation reveal? Rubric: 3 = new namespace missed; 2 = partial; 1 = vague; 0 = none.

Synthesis

Check the pack, validate in production and close gaps.

Continue studying — chapter 27 of 30

← Chapter 26 (draft) · Course chapter index · Chapter 28 (draft) →

Chapter 27 — Deployment and validation