Chapter 33 — OT, IoT and cyber-physical security: safety and resilience
Lesson map
Lesson 33 published. Map, example, EX097.
flowchart TD
N0["Learning outcomes"]
N1["33.1 Executable stops"]
N2["33.2 Rollback"]
N3["33.3 Tabletop"]
N4["Practice note"]
N5["33.1b Secondary contacts and resume authority"]
N6["Field note — resume token"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX097"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["33.1 Executable stops"]
N2["33.2 Rollback"]
N3["33.3 Tabletop"]
N4["Practice note"]
N5["33.1b Secondary contacts and resume authority"]
N6["Field note — resume token"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX097"]
[Learning outcomes]
|
v
[33.1 Executable stops]
|
v
[33.2 Rollback]
|
v
[33.3 Tabletop]
|
v
[Practice note]
|
v
[33.1b Secondary contacts and resume authority]
|
v
[Field note — resume token]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX097]
Learning outcomes
You will (1) write stop conditions as executable checks; (2) plan rollback of tool changes mid-window; (3) run a tabletop on a deny-list or cycle-time miss.
33.1 Executable stops
Stop conditions are code: cycle-time canary, process-alarm feed, client kill-switch file, SIS fault input.
At ABP, the serial monitor's stop logic reads the cycle-time canary, the process-alarm feed, a kill-switch file and SIS-B's fault input. Any one of them stops assessment activity at once. A human call to stop always wins, and the software has no setting that can override it or delay it until a running job finishes. Each stop the monitor logs records which input fired, so the hot review can tell a canary trip from a human call.
33.2 Rollback
Pin releases; keep previous monitor binary signed. If a new build mis-evaluates allow-lists, roll back within minutes.
Lena keeps the previous signed monitor build on the bench and in the window kit, so rollback is a tested command, not a rebuild. The rollback was rehearsed before the first plant window and is repeated after each major release. A rollback restores both the binary and the allow-list version that ran with it. Rollback restores the previous state within minutes. Both are pinned.
33.3 Tabletop
Walk a scenario: cycle time exceeds 50 ms at 02:40; who is called; what is said; when resume is allowed. Update the runbook where the tabletop stuck.
Synthetic case — Ashford Batch Plant. Tabletop found the kill-switch file path undocumented; runbook v1.4 fixed it before the next window.
ABP's tabletops run each term with the shift lead present, and every gap they expose is closed by a new runbook version before the next window. Each scenario names who is called, what they are told and who may allow work to resume.
33.4 Errors and limits
Resilience theatre is a binder nobody opens. Drill once per term.
Lena's resilience drills at ABP happen once per term, use the real runbook and include at least one stop and one rollback. Each drill produces at least one runbook edit or a recorded reason why none was needed. A binder that has not been opened since the last audit is treated as untested, however complete it looks on the shelf. Drill results are filed with the runbook version. Gaps get owners.
Practice note
Pre-write SMS/email templates for stop, resume and rollback. Under stress people omit the work-permit version or the next checkpoint time.
33.1b Secondary contacts and resume authority
A stop page that reaches only the window lead fails when the lead is out on the plant floor without signal. Maintain primary and secondary contacts with acknowledged handoff. Who may resume after a stop? Name the role in the work permit. The monitor should require a resume token tied to that role and AllowListVersion.
After every real stop, run a five-minute hotwash and file at least one runbook edit. Hotwash notes record the trigger, the time to stop and the time to resume, so drills can be compared with real stops.
Field note — resume token
After a stop, resume required a token issued to the process-safety contact's role and bound to AllowListVersion v1.4.2. A Boolean flip on the jumphost was removed from the UI. Resume without authority is how stop conditions become optional.
33.5 Worked example — an executable stop during commissioning (synthetic ABP)
During firewall cut-over on the L3–L2 conduit, the stop condition reads: "If the HMI loses PLC-A values for more than 10 seconds, or any SIS-B trip occurs, revert to the previous rule set." The revert is a single pre-staged command tested twice the day before, and the shift lead holds the authority to call it; the engineer does not argue. Resume requires a written token from the shift lead after the cause is understood. Separately, Lena confirms SIS-B's high-level trip on R-101 is hardwired and unaffected by the change, consistent with IEC 61511's expectation that the safety function stays independent.
Exercises
- EX097 — Kill switch. Design a human kill switch honored in ≤30s. Rubric: 3 = mechanism + poll + fail closed; 2 = idea; 1 = vague; 0 = none.
- EX098 — Rollback. Three preconditions to roll back mid-window. Rubric: 3 = prior binary, policy pin, notify process-safety; 2 = two; 1 = one; 0 = none.
- EX099 — Inject. One tabletop inject. Rubric: 3 = specific; 2 = vague; 1 = none; 0 = unsafe.
Synthesis
Stops the monitor can execute, rollbacks the site can perform, tabletops that update CNHP runbooks.
Continue studying — chapter 33 of 50
← Chapter 32 (draft) · Course chapter index · Chapter 34 (draft) →