Chapter 17 — Contained laboratory II | Technical Quantum and post-quantum security

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 17 — Contained laboratory II

Lesson map

Lesson 17 published. Map, example, EX049.

flowchart TD
  N0["Learning outcomes"]
  N1["17.1 Lab layout"]
  N2["17.2 Scenarios"]
  N3["17.3 Timing"]
  N4["17.4 Debrief"]
  N5["17.6 Hands-on activity"]
  N6["17.7 Working as a pair"]
  N7["17.9 Recording the exercise"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX049"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["17.1 Lab layout"]
  N2["17.2 Scenarios"]
  N3["17.3 Timing"]
  N4["17.4 Debrief"]
  N5["17.6 Hands-on activity"]
  N6["17.7 Working as a pair"]
  N7["17.9 Recording the exercise"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX049"]

[Learning outcomes]
    |
    v
[17.1 Lab layout]
    |
    v
[17.2 Scenarios]
    |
    v
[17.3 Timing]
    |
    v
[17.4 Debrief]
    |
    v
[17.6 Hands-on activity]
    |
    v
[17.7 Working as a pair]
    |
    v
[17.9 Recording the exercise]
    |
    v
[Worked example] --> [Exercise EX049]

Learning outcomes

You will (1) run a multi-hop lab with monitoring; (2) inject and diagnose failures; (3) practise runbooks under time pressure; and (4) debrief and improve documents.

17.1 Lab layout

Client → reverse proxy → web server, plus an SSH server, all hybrid-capable; a log parser feeding a simple dashboard; a "legacy" client; and a proxy that can be set to drop large first messages.

Build it from the Chapter 12 lab, adding the proxy, the log parser and the dashboard. Record all versions. The proxy that drops large messages reproduces HHG's Summit problem safely. Snapshot the lab in its working state, so you can restore it between scenarios. Label every lab host clearly as lab.

17.2 Scenarios

The instructor (or a teammate) injects one failure at a time: reorder groups on the proxy; restrict the proxy buffer; remove classical fallback on SSH; replace the server certificate with one from an untrusted lab root. You diagnose with logs and tools and fix using runbooks.

Each scenario matches a failure mode from Chapter 8: silent fallback, a large message dropped, no shared group and a certificate failure. Record the symptoms you see, the runbook you used and whether it worked. Restore the lab to its baseline before the next scenario.

17.3 Timing

Record time to detect, time to diagnose and time to fix. The goal is not speed for its own sake but finding which steps of the runbook slowed you down.

Measure from when the failure was injected to when you noticed it, then to when you knew the cause, then to when service was restored and verified. A long diagnosis time usually means a missing check or a confusing step in the runbook. Compare times across sessions. Note which step took longest.

17.4 Debrief

After each scenario: what happened, what you did, what helped, what to change in the runbook or dashboard. Update the documents immediately.

At HHG, the debrief after the proxy buffer scenario added a step and a dashboard panel. Keep debriefs short and blame-free, and focus on documents and tools rather than on who was slow. Record each change with the scenario that caused it. Test each updated runbook in the next session to confirm it helps. Share the changes with the team.

17.5 Worked example

Scenario 2 (proxy buffer): detection 3 minutes (timeouts in logs), diagnosis 11 minutes (operator first suspected the server), fix 2 minutes (proxy setting restored). Debrief: add "check proxy before server when timeouts appear on one path" to the runbook; add per-path panels to the dashboard.

The debrief turned an 11-minute diagnosis into two concrete improvements. The operator suspected the server first, which cost time; the new runbook step and per-path panels make the proxy the first thing checked. Record the times so later sessions can show whether the changes helped.

17.6 Hands-on activity

Run all four scenarios; record times and debrief notes.

Run the four scenarios in turn: reordered groups on the proxy, a restricted proxy buffer, removed classical fallback on SSH and an untrusted lab root on the server. For each, record times to detect, diagnose and fix, the runbook used, and what slowed you down. Restore the lab between scenarios. End with a list of changes to runbooks or the dashboard, and make them before the next session. Share the list with your instructor.

17.7 Working as a pair

Practise in pairs: one operator drives, the other reads the runbook and records times and observations. Swap roles each scenario. Pairing exposes unclear steps quickly and mirrors how incidents are handled on real on-call rotations.

Record observations from both roles in the debrief. The reader should follow the runbook exactly as written, so unclear or missing steps become visible rather than being filled in from memory. Agree before each scenario who drives and who reads, and record any step the reader had to explain.

17.8 Common errors and limits

Skipping debriefs.
Fixing without diagnosing.
Not updating documents after lessons.

Each error wastes the exercise. Skipping debriefs means the same steps slow you down next time. Fixing without diagnosing, for example restoring the proxy setting without checking the logs, teaches nothing and may hide a second fault. Not updating documents leaves the lesson in your head instead of the runbook, where the night shift needs it. Check each session for all three. Fix any you find. Note what you changed in each runbook after the session.

17.9 Recording the exercise

Keep a short log of each lab session — date, scenarios, times, debrief actions and who took part — in the team repository. Over several sessions the log shows whether detection and fix times are improving and which runbooks have changed as a result. It is also evidence for supervisors and auditors that the team practises its procedures rather than merely writing them.

Keep the log factual and brief. Review it each quarter. Share a summary with the team after each session, so lessons reach operators who did not take part.

Exercises

EX049 — Scenario plan. Design a fifth scenario for the lab. Rubric: 3 = realistic injection, expected symptom, fix.
EX050 — Timing analysis. In the worked example, which phase took longest and how would you shorten it? Rubric: 3 = diagnosis; runbook ordering/panels.
EX051 — Certificate scenario. What symptom and fix do you expect for the untrusted lab root scenario? Rubric: 3 = verification error; add root or restore trusted cert.

Synthesis

Multi-hop labs with injected failures and timed, debriefed runbook practice build operators who can find and fix problems quickly and improve their tools.

Continue studying — chapter 17 of 18

← Chapter 16 (draft) · Course chapter index · Chapter 18 (draft) →