Chapter 20 — OT, IoT and cyber-physical security: troubleshooting
Lesson map
Lesson 20 published. Map, example, EX058.
flowchart TD
N0["Learning outcomes"]
N1["20.1 Most faults are not attacks"]
N2["20.2 A layered method"]
N3["20.3 Evidence-preserving habits"]
N4["20.4 Stop and escalate"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX058"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["20.1 Most faults are not attacks"]
N2["20.2 A layered method"]
N3["20.3 Evidence-preserving habits"]
N4["20.4 Stop and escalate"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX058"]
[Learning outcomes]
|
v
[20.1 Most faults are not attacks]
|
v
[20.2 A layered method]
|
v
[20.3 Evidence-preserving habits]
|
v
[20.4 Stop and escalate]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX058]
Learning outcomes
You will (1) separate cyber causes from ordinary faults; (2) troubleshoot communication failures methodically; (3) avoid troubleshooting steps that destroy evidence; and (4) know when to stop and escalate.
20.1 Most faults are not attacks
When HMI-C shows stale values, the cause is usually a cable, a switch port, a failed power supply or an expired licence. Treat it as a fault first, but keep the security possibility in mind and preserve evidence while you work. Jumping to "we are hacked" wastes time; ignoring the possibility can destroy the record an investigation would need.
Priya's first stale-value fault on PLC-R2 was a patch cable with rising CRC errors; the second, a week later, was not.
20.2 A layered method
Work up the stack: power and link lights; switch port status and errors; IP reachability from a host on the same subnet (with permission); the protocol session (does HMI-C show a communication error code?); then application configuration. Change one thing at a time and write down each step and result.
Priya's first PLC-R2 fault followed this order: link light on, switch port showing rising CRC errors, new patch cable, errors stop, values update. She recorded the fault as physical, with the error counters as evidence. Stopping at the first layer that explained the symptom kept her from touching the PLC at all.
20.3 Evidence-preserving habits
Before rebooting a host that might be compromised, export its logs and note its running processes. Before replacing a device, photograph its labels and settings. A reboot clears memory that might show malware; a replacement removes the original firmware state.
When PLC-R2's values froze again a week later with no port errors, Priya compared its program against the backup before anyone restarted it, and found a difference. A restart would have run the same changed program again, so it would not have fixed anything, and it would have cleared the controller's runtime state and diagnostic buffer. The usual next step after a failed restart, downloading the backup, would have overwritten the changed program, which was the main evidence of who changed it and when. She escalated with the comparison output and the port counters attached.
20.4 Stop and escalate
Stop and call the security lead if you find: unknown processes or accounts, configuration changes nobody made, traffic to unknown destinations, or a device whose program differs from the backup.
At NCS, the second PLC-R2 freeze met the fourth condition: the controller's program differed from the backup, so Priya escalated instead of restarting. The same rule applied on her third site visit, when an operator asked her to "quickly reset" a frozen HMI. She declined, checked the switch port counters read-only and called the contractor, who found a failing power supply. If the freeze had had a cyber cause, the reset would have erased what the investigation needed.
Worked example (synthetic NCS)
HMI-C shows stale PLC-R2 values. Priya checks: link light on; switch port shows rising CRC errors; a new patch cable fixes the errors and the values update. She records the fault as physical, with the error counters as evidence. A week later, PLC-R2's values freeze again with no errors on the port; this time the PLC's program comparison against the backup shows a difference, and she escalates instead of restarting the controller.
Synthetic case — Northgate Cold Storage. Both events are invented.
Errors and limits
Pinging fragile devices repeatedly can itself cause problems; keep checks light. Troubleshooting on live controllers belongs to the contractor or plant engineer.
Exercises
- EX058 — First three checks. List the first three checks for stale HMI values. Rubric: 3 = power/link, port status/errors, protocol error on HMI (in order); 2 = two; 1 = reboot first; 0 = none.
- EX059 — Before rebooting. What two things do you capture before rebooting a possibly compromised HMI-C? Rubric: 3 = logs export and running processes/connections; 2 = one; 1 = vague; 0 = none.
- EX060 — The escalation trigger. Which observation in the second event triggered escalation and why? Rubric: 3 = program differs from backup without a change record; possible tampering; 2 = observation only; 1 = errors; 0 = none.
Synthesis
Troubleshoot methodically, assume faults first, preserve evidence while working, and stop the moment something has changed that nobody changed.
Continue studying — chapter 20 of 30
← Chapter 19 (draft) · Course chapter index · Chapter 21 (draft) →