Chapter 10 — Critical infrastructure and sector security: quality and troubleshooting
Lesson map
Lesson 10 published. Map, example, EX046.
flowchart TD
N0["Learning outcomes and connection"]
N1["10.1 How programmes fail"]
N2["10.2 Troubleshooting a coordination failure"]
N3["10.3 Decision quality"]
N4["10.5 Closing corrective actions"]
N5["10.6 Practical activity"]
N6["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX046"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["10.1 How programmes fail"]
N2["10.2 Troubleshooting a coordination failure"]
N3["10.3 Decision quality"]
N4["10.5 Closing corrective actions"]
N5["10.6 Practical activity"]
N6["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX046"]
[Learning outcomes and connection]
|
v
[10.1 How programmes fail]
|
v
[10.2 Troubleshooting a coordination failure]
|
v
[10.3 Decision quality]
|
v
[10.5 Closing corrective actions]
|
v
[10.6 Practical activity]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX046]
Learning outcomes and connection
After this chapter you should be able to:
- recognise the common failure patterns of security coordination programmes;
- troubleshoot a coordination failure methodically — symptom, hypotheses, evidence, cause, fix — instead of blaming people;
- measure the quality of decisions in an exercise or incident, not only their speed;
- run a corrective-action process that actually closes actions.
Chapter 9 exposed real coordination failures. This chapter treats the programme itself as a system that can be diagnosed and improved.
10.1 How programmes fail
Most coordination failures fall into a few patterns:
- Ownership gaps — activities or risks with no accountable person (Chapters 4 and 6).
- Duplicated responsibility — two people each think the other acted.
- Stale information — contacts, dependency maps, supplier lists or registers that were right last year.
- Paper controls — documented but never observed or tested (Chapter 7).
- Bottlenecks — one person on every critical path.
- Action decay — lessons identified after every exercise, never implemented.
- Speed without quality — decisions taken on time but without rationale, wrong information or the right people.
10.2 Troubleshooting a coordination failure
Use the same discipline engineers use for technical faults:
- State the symptom precisely ("the hospital was called 15 minutes after the contractual 2 hours").
- List hypotheses ("nobody owned the call"; "contact number wrong"; "the 2-hour clock started at a different moment for different people").
- Gather evidence — decision logs, call records, interviews — without blame.
- Eliminate hypotheses the evidence contradicts.
- Fix the cause, not the symptom, and decide how you will verify the fix.
A useful habit: ask "what made this the reasonable thing to do at the time?" People rarely act carelessly in a crisis; they act on the information and roles they had.
10.3 Decision quality
Timeliness is easy to measure; quality is harder but more important. For each key decision in an exercise or incident, check: Did it have an owner? Was it taken within the target time? Was the rationale recorded? Were the people who needed to be consulted, consulted? Was it based on verified information, with assumptions labelled? NIST's guide to test, training and exercise programmes describes evaluating exercises against objectives and capturing lessons (NIST SP 800-84).
10.4 Worked example: the May tabletop log (synthetic)
In the May tabletop exercise DRY CREEK (its design is the worked example of Chapter 11), an evaluator kept a decision log. Lab 10 checks it.
Lab 10 (save as lab10.py, run with python3 lab10.py):
# Lab 10 - tabletop decision-log quality check (offline, standard library only; synthetic exercise log)
import csv, io
LOG = """minute,inject,decision,owner,target_min,taken_min,recorded_rationale
0,I1 HMI shows unexpected setpoint change,convene incident team,Coordinator,15,9,yes
20,I2 vendor reports stolen laptop,suspend vendor remote access,Ops manager,30,52,yes
35,I3 station KW-04 dosing alarm,switch KW-04 to manual,Ops manager,10,6,no
50,I4 journalist calls,holding statement,,30,41,yes
65,I5 hospital asks about water safety,brief hospital,Coordinator,20,,no
80,I6 regulator early-warning clock,send early warning,Coordinator,60,44,yes
"""
rows = list(csv.DictReader(io.StringIO(LOG)))
met = 0
for r in rows:
f = []
if not r["owner"]: f.append("no owner")
if not r["taken_min"]: f.append("no decision taken")
elif int(r["taken_min"]) > int(r["target_min"]): f.append(f"late {int(r['taken_min'])-int(r['target_min'])} min")
else: met += 1
if r["recorded_rationale"] != "yes": f.append("no rationale recorded")
print(f"{r['inject'][:2]} {r['decision']:30s} {'ok' if not f else '; '.join(f)}")
print(f"decisions within target: {met}/{len(rows)}")
Expected output (Python 3.13.5):
I1 convene incident team ok
I2 suspend vendor remote access late 22 min
I3 switch KW-04 to manual no rationale recorded
I4 holding statement no owner; late 11 min
I5 brief hospital no decision taken; no rationale recorded
I6 send early warning ok
decisions within target: 3/6
The headline "3/6 within target" hides more than it shows:
- I3's decision to switch KW-04 to manual was fast (6 minutes against 10) and counted as on time, but no rationale was recorded. For a safety-related decision, that matters: in a real incident, the review would not know whether it was taken because of dosing data, a precaution or a guess.
- I4's holding statement had no owner — the communications defect from Chapter 6 again — and was late.
- I5, the hospital briefing, was never decided. In the hot wash, players said they "assumed comms had it".
- I2, suspending vendor access, was 22 minutes late. Players explained that nobody knew whether suspending HydroLogic's access would stop support for a dosing problem then in progress. That is a missing information cause, not slowness.
The coordinator's troubleshooting table:
| Symptom | Hypotheses | Evidence | Cause | Fix | Verify by |
|---|---|---|---|---|---|
| I5 hospital briefing not done | No owner; contact unknown; forgotten under load | Hot wash: "assumed comms had it"; RACI had two R | Duplicated responsibility | Single lead in RACI; hospital call on the incident checklist | Next exercise inject on hospital |
| I2 vendor suspension late | Approval unclear; impact unknown | Players asked who depends on HydroLogic access | Missing dependency information | Add "who depends on this supplier's access" to supplier register | Lab 8 review includes dependency column |
| I3 no rationale | Log template lacks field; time pressure | Template has no rationale column | Tool design | Add rationale field; scribe role | Lab 10 rerun shows rationale |
10.5 Closing corrective actions
Lessons do not improve anything until they become actions with owners, dates and verification — and are closed. Track them in one place, review them at the risk committee, and verify each closure with evidence (the next exercise, a test, a document). A useful programme metric is the percentage of exercise actions closed and verified within their due date.
In the DRY CREEK review, each fix has a matching verification: the next exercise inject on the hospital, a supplier register with a dependency column checked in the next access review, and a rerun of Lab 10 showing recorded rationales.
10.6 Practical activity
Take a recent failure in any team you know (a missed deadline, a duplicated task). Fill the troubleshooting table without naming individuals and propose a fix with a verification method.
Write the symptom as a precise fact with times or counts, as in the hospital example, rather than a judgement. List at least three hypotheses before looking for evidence, and note which evidence ruled each one out. Ask what made the action reasonable for the people involved at the time. Then choose a fix that changes a role, a tool or information, not one that asks people to try harder, and say how you will know it worked, for example in the next exercise or review.
10.7 Common errors and limits
- Concluding "human error" and stopping.
- Actions without verification, closed on the owner's word.
-
Over-correcting with heavy process that slows real responses; keep fixes proportionate.
-
Fixing the symptom with an exhortation ("communicate better"), such as reminding staff to call the hospital, while the RACI still lists two responsible people.
- Reporting exercise success as speed alone, counting a fast decision as good when no rationale was recorded, as with the KW-04 switch to manual in DRY CREEK.
- Tracking actions in several places so that nobody can see which are overdue.
Exercises — chapter 10
EX046 — Read Lab 10. Objective: interpret decision quality. Inputs: Lab 10 output. Steps: explain why "3/6 within target" overstates performance and name the most serious defect. Expected evidence: explanation. Rubric: 3 = I3 counted on time without rationale; I5 not taken at all (critical-customer risk); I4 unowned; most serious is I5 or I3 with reasoning; 2 = lists defects without judgement; 1 = accepts 3/6; 0 = none.
EX047 — Add a quality rule. Objective: encode another check. Inputs: Lab 10. Steps: add safety and consulted columns; flag any safety-related decision whose owner is not the operations manager and where the operations manager was not consulted; test with the six original rows plus one invented row in which the security lead decides to isolate a station without consulting operations; show the output. Expected evidence: code and output. Rubric: 3 = columns populated for all rows, rule restricted to safety decisions, the invented row flagged and I3 (owned by operations) not flagged, with that result explained; 2 = rule flags every row lacking a consulted entry; 1 = rule only; 0 = none.
EX048 — Troubleshoot I4. Objective: apply 10.2. Inputs: I4 row and Chapter 6 RACI. Steps: fill the troubleshooting table for the holding statement. Expected evidence: one table row with hypotheses. Rubric: 3 = at least two hypotheses, evidence (RACI had no A for public statement), cause, fix (CEO A, comms R, template), verification (next exercise); 2 = no verification; 1 = "comms were slow"; 0 = none.
EX049 — Diagnose stale information. Objective: recognise decay. Inputs: in an incident, the hospital's out-of-hours number reached a disconnected line. Steps: propose a cause analysis and a durable fix. Expected evidence: analysis and fix. Rubric: 3 = cause: no owner or schedule for contact verification; fix: owned quarterly test calls, update on hospital reorganisations, second contact route; verification metric; 2 = one-off update; 1 = blame; 0 = none.
EX050 — Close actions. Objective: design corrective-action tracking. Inputs: the three fixes in 10.4. Steps: write an action log with owner, due date, evidence of closure and verification method. Expected evidence: log. Rubric: 3 = each has an owner with authority, realistic date, closure evidence distinct from owner's statement, and verification in a later exercise or review; 2 = no verification; 1 = list only; 0 = none.
Chapter synthesis
Coordination programmes fail in recognisable ways — ownership gaps, duplicated responsibility, stale information, paper controls, bottlenecks, action decay and speed without quality. Troubleshoot them like faults, from symptom through hypotheses and evidence to cause, without blame. Measure decision quality, not just speed, and close corrective actions only with verified evidence.
Continue studying — chapter 10 of 12
← Chapter 09 · Course chapter index · Chapter 11 →
Continue to the next authored lesson.