Chapter 37 — OT, IoT and cyber-physical security: operations
Lesson map
Lesson 37 published. Map, example, EX109.
flowchart TD
N0["Learning outcomes"]
N1["37.1 Runbook contents"]
N2["37.2 Handover"]
N3["37.3 Close-out"]
N4["Practice note"]
N5["37.1b Calendar and spare kits"]
N6["Field note — offline kit"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX109"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["37.1 Runbook contents"]
N2["37.2 Handover"]
N3["37.3 Close-out"]
N4["Practice note"]
N5["37.1b Calendar and spare kits"]
N6["Field note — offline kit"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX109"]
[Learning outcomes]
|
v
[37.1 Runbook contents]
|
v
[37.2 Handover]
|
v
[37.3 Close-out]
|
v
[Practice note]
|
v
[37.1b Calendar and spare kits]
|
v
[Field note — offline kit]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX109]
Learning outcomes
You will (1) write a window runbook; (2) hand over between shifts; (3) manage close-out including account disable and evidence seal.
37.1 Runbook contents
Contacts, work-permit pointer, tool versions, stop conditions, escalation tree, evidence paths. Keep it short enough to use under stress.
Lena's ABP runbook fits on four printed pages. It names the window lead and the control engineer, links the work permit and lists the versions of each tool allowed. Stop conditions, including cycle time at 50 ms, are on the first page, followed by the escalation tree and the location of the evidence folder. Runbook v1.4 added the kill-switch file path after the tabletop found it missing, and every change since is recorded in the version history.
37.2 Handover
Shift log: open findings, last event ids, pending approvals, kill-switch state, process alarms during the window. Verbal handover plus the log.
At ABP the outgoing lead reads the shift log aloud to the incoming lead and both initial it. The log lists open findings with owners, the last event ids recorded by the monitor, any approvals still pending, whether the kill switch is armed, and any process alarms raised during the window. Speaking the items aloud catches entries one person thought obvious and the other had never seen, such as a temporary rule awaiting removal.
37.3 Close-out
Disable temporary accounts, rotate secrets, seal evidence, export metrics, schedule retrospective. A checklist beats memory.
"""Lab: fit patch jobs into plant maintenance windows (synthetic, greedy by risk)."""
windows = [("W1 Sat 02-06", 4.0), ("W2 Sun 02-04", 2.0)]
jobs = [("HMI OS update", 1.5, 8), ("historian DB patch", 2.5, 5), ("PLC-A firmware", 3.0, 9),
("eng-WS AV update", 0.5, 3), ("switch firmware", 1.0, 6)] # name, hours, risk score
left = {w: h for w, h in windows}; plan = {w: [] for w, _ in windows}; deferred = []
for name, h, r in sorted(jobs, key=lambda j: -j[2]):
for w, _ in windows:
if left[w] >= h:
left[w] -= h; plan[w].append(name); break
else:
deferred.append(name)
for w, _ in windows:
print(w, plan[w], f"spare={left[w]:.1f}h")
print("deferred:", deferred)
Expected output:
W1 Sat 02-06 ['PLC-A firmware', 'switch firmware'] spare=0.0h
W2 Sun 02-04 ['HMI OS update', 'eng-WS AV update'] spare=0.0h
deferred: ['historian DB patch']
Synthetic case — Ashford Batch Plant. Close-out caught the deferred historian patch still without a date — escalated before the retro.
Lena's close-out checklist is run by two people and its result is stored with a hash. A historian patch had been deferred during the window with no date, so the checklist flagged it and the patch received an owner and a date before the retrospective met.
37.4 Errors and limits
Runbooks in wikis nobody can access from the control room fail; keep an offline copy.
At ABP the control room lost network access to the wiki during one window, and the lead needed the escalation tree immediately. Since then, the latest runbook is printed and kept in a folder beside HMI-1, and a copy sits on the engineering workstation for offline use. Each printed copy carries its version number, and the close-out checklist confirms the paper copy matches the current version, so an old procedure is not followed by mistake during an incident.
Practice note
On-call for CNHP tools is not a license to widen conduits. Pages are about tool safety and permit enforcement; process incidents route to the plant IR/process-safety path.
37.1b Calendar and spare kits
Windows belong on a shared calendar with work-permit version in the description. Accidental overlap of two noisy methods on the same trunk is an operations failure. Ops kits include spare jump-host images, printed contact cards and offline permit copies for when SSO is down.
Close-out is not marked done until both checklist signers are recorded by name. Handover includes reading the last ten monitor denies aloud to the incoming lead.
Lena checks the calendar at every handover. Overlaps are logged as findings.
Field note — offline kit
SSO failed during a Sunday window. The offline runbook copy and printed contact cards let the team stop cleanly and notify process safety. Since then the kit is checked at every close-out.
Operations metrics include pages per window and mean time to acknowledge stops; vanity metrics like hosts touched stay off the ops dashboard.
37.5 Worked example — a shift handover with security content (synthetic ABP)
ABP's handover sheet gains three lines: open CNHP alerts and who owns them; any vendor remote session enabled or scheduled; and any temporary rule or bypass in force with its expiry time. On 14 March the night shift recorded a temporary firewall rule for a vendor diagnostic expiring at 06:00; the day shift checked at 06:15, found it still present, and removed it with the engineer's agreement. Without the handover line, the rule would have become a permanent conduit. Close-out of each security work order requires the evidence pointer and confirmation that temporary changes were reverted.
Exercises
- EX109 — Checklist. Six close-out steps. Rubric: 3 = six sensible; 2 = four; 1 = two; 0 = none.
- EX110 — Handover. Three fields the shift log must include. Rubric: 3 = findings/events/approvals (or equiv.); 2 = two; 1 = one; 0 = none.
- EX111 — Lab. Which job was deferred and what must close-out record? Rubric: 3 = historian patch + residual risk + next window date; 2 = name only; 1 = vague; 0 = wrong.
Synthesis
Operate with runbooks, handovers and ruthless close-out.
Continue studying — chapter 37 of 50
← Chapter 36 (draft) · Course chapter index · Chapter 38 (draft) →