Chapter 16 — Technical documentation and safety
Lesson map
Lesson 16 published. Map, example, EX046.
flowchart TD
N0["Learning outcomes"]
N1["16.1 Runbooks"]
N2["16.2 Baselines"]
N3["16.3 Safety and authorization"]
N4["16.4 Keeping documents current"]
N5["16.6 Hands-on activity"]
N6["16.7 Writing for the night shift"]
N7["16.9 Sharing documentation"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX046"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["16.1 Runbooks"]
N2["16.2 Baselines"]
N3["16.3 Safety and authorization"]
N4["16.4 Keeping documents current"]
N5["16.6 Hands-on activity"]
N6["16.7 Writing for the night shift"]
N7["16.9 Sharing documentation"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX046"]
[Learning outcomes]
|
v
[16.1 Runbooks]
|
v
[16.2 Baselines]
|
v
[16.3 Safety and authorization]
|
v
[16.4 Keeping documents current]
|
v
[16.6 Hands-on activity]
|
v
[16.7 Writing for the night shift]
|
v
[16.9 Sharing documentation]
|
v
[Worked example] --> [Exercise EX046]
Learning outcomes
You will (1) write runbooks for post-quantum operations; (2) document configuration baselines; (3) follow safety and authorization rules; and (4) keep documentation current.
16.1 Runbooks
A runbook says: when to use it (symptoms), what to check, safe actions, rollback, how to confirm, who to call. Write runbooks for hybrid share drops, large-handshake timeouts, no shared group, and certificate verification errors.
At HHG, the hybrid share drop runbook is an example. Keep each runbook short enough to follow during an outage, and test it in the lab before relying on it. Name an owner for each runbook and the date it was last reviewed. Link each to the alerts that should trigger it.
16.2 Baselines
A configuration baseline lists the approved settings for each service type: TLS versions, groups and order, SSH KexAlgorithms, certificate types, logging fields. Store it in version control; changes go through review.
At HHG, the baseline holds the approved group order X25519MLKEM768:X25519:secp256r1 for TLS and the hybrid-first KexAlgorithms for SSH. The booking API rollback used it. Compare running configurations with the baseline regularly and treat any difference as a finding. Record each change to the baseline with its reason.
16.3 Safety and authorization
Never run active tests on production without written approval; never try to force downgrades outside the isolated lab; never handle keys outside approved tools; report exposed keys immediately. Label synthetic data as such in all documents and training material.
At HHG, size-failure tests were done in the isolated lab, not on Summit's production firewall. If a task seems to need an active test on production, stop and ask for written approval first, and keep the approval with the evidence. Report breaches of these rules openly.
16.4 Keeping documents current
Review runbooks after every incident and every major change. Put a "last reviewed" date and owner at the top. Remove steps that no longer apply.
At HHG, the Summit incident should lead to a review of the large-handshake runbook. Record each change with a reason, so readers know why a step exists. A document nobody has reviewed for a year should be treated as unreliable until checked. Assign each review to a named owner and record the date it was done.
16.5 Worked example
HHG runbook "Hybrid share drop on a TLS service": symptoms — dashboard alert, hybrid share below 50 %; checks — last change on the service, current TLS profile vs baseline; actions — re-apply baseline profile on the affected listener; rollback — previous profile from version control; confirm — hybrid share recovers within 30 minutes; call — Web Ops on-call, then platform lead. Last reviewed 2 October; owner Web Ops.
The runbook is complete: each part answers a question the on-call operator will have, in order. The 30-minute confirmation gives a clear stop point.
16.6 Hands-on activity
Write a runbook for "No shared group on SFTP" using the format above.
Use the same headings: symptoms, checks, actions, rollback, confirmation and who to call. Base the checks on what you learned: the server's KexAlgorithms against the baseline, the client's version and the last change. Keep the action narrow, for example restoring classical fallback on the SFTP server only. Add an owner and a review date, and test the runbook in the lab before submitting it. Ask a peer to follow it.
16.7 Writing for the night shift
Runbooks are read under pressure, often at night. Use short numbered steps, exact commands, expected outputs and clear stop points ("if the share has not recovered after step 4, call the platform lead"). Avoid long explanations in the steps; link to background instead.
At HHG, the hybrid share drop runbook gives the 30-minute confirmation point and names whom to call next. Test each runbook with someone who did not write it, ideally in the lab at the end of a shift, and fix any step that made them hesitate.
16.8 Common errors and limits
- Runbooks without rollback.
- Baselines kept only in someone's head.
- Out-of-date documents.
Each error leaves the team exposed. A runbook without rollback leaves the on-call operator stuck when a fix fails. A baseline kept in someone's head disappears when they are away, and settings drift. An out-of-date document can direct people to steps that no longer work or are unsafe. Check each runbook and baseline for all three at every review. Correct any you find. Record when each was last checked and by whom.
16.9 Sharing documentation
Store runbooks and baselines where on-call staff can reach them during an outage — not only in a system that might itself be down. Many teams keep a read-only copy in a second location. Mark documents that contain sensitive configuration detail and restrict them accordingly, while keeping general procedures widely readable.
Review access to restricted documents when staff change roles. At HHG, the hybrid share drop runbook would be needed during a load-balancer outage, so a copy must be reachable without it. Test access to the second location twice a year.
Exercises
- EX046 — Runbook. Write the "no shared group" runbook. Rubric: 3 = symptoms, checks, actions, rollback, confirm, contacts.
- EX047 — Baseline. List five items in a TLS baseline. Rubric: 3 = versions, groups/order, cert types, logging, cipher suites.
- EX048 — Safety rule. Explain why downgrade tests belong only in the lab. Rubric: 3 = risk to production sessions, authorization.
Synthesis
Runbooks, baselines and safety rules turn individual skill into reliable team operations; documents stay useful only when owned and reviewed.
Continue studying — chapter 16 of 18
← Chapter 15 (draft) · Course chapter index · Chapter 17 (draft) →