Chapter 13 — Critical infrastructure and sector security: troubleshooting
Lesson map
Lesson 13 published. Map, example, EX049.
flowchart TD
N0["Learning outcomes and connection"]
N1["13.1 The method"]
N2["13.2 From fault to incident"]
N3["13.4 Second case: the corporate desk at KW-04"]
N4["13.5 The troubleshooting log"]
N5["13.6 Practical activity"]
N6["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX049"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["13.1 The method"]
N2["13.2 From fault to incident"]
N3["13.4 Second case: the corporate desk at KW-04"]
N4["13.5 The troubleshooting log"]
N5["13.6 Practical activity"]
N6["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX049"]
[Learning outcomes and connection]
|
v
[13.1 The method]
|
v
[13.2 From fault to incident]
|
v
[13.4 Second case: the corporate desk at KW-04]
|
v
[13.5 The troubleshooting log]
|
v
[13.6 Practical activity]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX049]
Learning outcomes and connection
After this chapter you should be able to:
- apply a structured troubleshooting method — define the symptom, collect read-only observations, list hypotheses, test predictions, preserve, then change;
- eliminate hypotheses by comparing what each predicts with what you observed, in a way others can check;
- recognize when a fault investigation becomes a security incident, and what to preserve at that moment;
- keep a troubleshooting log that is also evidence.
Chapter 8 described symptoms; Chapters 7, 9 and 11 gave you observations. This chapter puts them together under time pressure, when a station has gone quiet and operators want it back.
13.1 The method
- Define the symptom precisely: which tag, since when (UTC), how detected, what still works.
- Collect read-only observations in order of cost: other stations' status; link and router counters; firewall logs; a TCP connect test from the normal polling source; a protocol read; device configuration records.
- List hypotheses — at least one ordinary and one security-relevant — before testing any.
- Write each hypothesis's predictions: what would you observe if it were true?
- Compare predictions with observations. A hypothesis contradicted by a reliable observation is refuted; one consistent with all of them stays.
- Preserve anything that might be evidence before changing it (Chapter 5).
- Change one thing, under a change record, and verify (Chapter 11).
- Record the cause, the evidence, and what would have detected it sooner.
The discipline of writing predictions before comparing is what stops the first plausible story from becoming the conclusion.
13.2 From fault to incident
A troubleshooting exercise becomes a possible incident when an observation is unexplained by any approved activity: an unapproved configuration session, an unknown source, a changed file with no change record. At that point stop changing things, preserve (configuration, logs, session records), and hand over through the incident process. NIST's incident response recommendations place this decision within the organization's broader risk management (SP 800-61 Rev. 3). The technician's job is not to decide that an attack happened; it is to notice that the explanation is missing and to keep the evidence intact.
13.3 Worked example: KW-09 goes quiet (synthetic)
At 07:05Z on 12 October, KW-09's readings stop arriving; all other stations are fine. You collect read-only observations for fifteen minutes and list five hypotheses.
Lab 13 (save as lab13.py, run with python3 lab13.py):
# Lab 13 - eliminate hypotheses for "KW-09 readings missing" (offline, standard library only)
# Synthetic observations collected read-only between 07:05 and 07:20Z on 2026-10-12.
observed = {
"rtu_ping": True, # RTU answers ICMP from the SCADA server
"tcp502_handshake": True, # TCP connect to 10.10.9.2:502 succeeds from SCADA server
"modbus_reply": "exception 2", # read of registers 0..3 returns illegal data address
"fw_denies": False, # no fw-01 denies for 10.10.9.2 in the window
"radio_errors": False, # cellular router CRC/error counters flat
"rtu_config_changed": True, # rtu-kw09 config session at 06:41Z (clock-corrected) by user eng
"change_ticket": False, # no approved change ticket for KW-09 today
"other_stations_ok": True,
}
hypotheses = { # what each hypothesis predicts we would observe
"H1 radio/link failure": {"rtu_ping": False, "tcp502_handshake": False, "radio_errors": True},
"H2 firewall change blocks polls": {"tcp502_handshake": False, "fw_denies": True},
"H3 SCADA server fault": {"other_stations_ok": False},
"H4 RTU register map changed": {"rtu_ping": True, "tcp502_handshake": True, "modbus_reply": "exception 2",
"rtu_config_changed": True},
"H5 RTU powered off": {"rtu_ping": False},
}
for h, pred in hypotheses.items():
clash = [k for k, v in pred.items() if observed[k] != v]
print(f"{h:<32} {'REFUTED by ' + ', '.join(clash) if clash else 'CONSISTENT with all observations'}")
print("unapproved RTU configuration session -> preserve RTU config and session logs, then treat as possible incident"
if observed["rtu_config_changed"] and not observed["change_ticket"] else "change was approved")
Expected output (Python 3.13.5):
H1 radio/link failure REFUTED by rtu_ping, tcp502_handshake, radio_errors
H2 firewall change blocks polls REFUTED by tcp502_handshake, fw_denies
H3 SCADA server fault REFUTED by other_stations_ok
H4 RTU register map changed CONSISTENT with all observations
H5 RTU powered off REFUTED by rtu_ping
unapproved RTU configuration session -> preserve RTU config and session logs, then treat as possible incident
Four hypotheses are refuted by specific observations: the link is up (ping, handshake, flat error counters); the firewall is not denying; the SCADA server is polling other stations successfully; the RTU is powered. One remains consistent: the RTU's register map changed, so the SCADA server's reads now hit addresses the device no longer has — exception 02, exactly as in the Chapter 12 lab. And the RTU logged a configuration session at 06:41Z (clock-corrected) with no change ticket.
That last observation turns a fault into a possible incident. The correct next steps are: preserve the RTU's current configuration export and its session log, and the jump-host and engineering-workstation logs for 06:30–06:50Z; inform the Kestrel Water duty manager that KW-09 values are unavailable (they may need a site visit for manual readings); open an incident record. Restoring the previous configuration is the fix — but only after preservation and with the incident lead's agreement.
13.4 Second case: the corporate desk at KW-04
Chapter 10 left an open item: fw-01 denies corporate-desk traffic to stations, yet a capture at KW-04 showed writes from 10.30.3.7. Hypotheses: (a) the capture's source address was translated by some device; (b) another path to KW-04 exists that bypasses fw-01. Read-only checks: the KW-04 cellular router's configuration export shows a port-forward of tcp/502 from its carrier-side private address, and the carrier's private network is also reachable from the corporate remote-access VPN. Hypothesis (b) is consistent; the path was installed during a 2024 commissioning and never removed. The lesson is architectural (Chapter 15): every station router is a potential boundary, and the model in Chapter 6 must include it.
13.5 The troubleshooting log
Write as you go, in UTC, with the four labels: observed, inferred, done, open. A good log lets a colleague taking over at shift change continue without asking you anything, and lets an investigator later reconstruct what was changed and when. Chapter 16 gives a template; Chapter 18 seals it as project evidence.
Include evidence references for each observation, such as the RTU config export hash and the jump-host log interval. In Lab 18 the log becomes 05-troubleshooting-log.md, and its manifest hash verifies OK when the package is checked.
13.6 Practical activity
Add hypothesis H6 "RTU firmware crashed into a degraded mode" to Lab 13 with predictions you think it makes, and observe whether the existing observations refute it. If they do not, name the single additional read-only observation that would.
If the existing observations do not refute H6, name one additional read-only check, such as a vendor diagnostic that reports firmware state, that would. Write the predictions before you invent the check, so the new observation has something to contradict. Keep H6 in the list until it is refuted or the incident process takes over.
13.7 Common errors and limits
- Treating a consistent hypothesis as proven; it is only not yet refuted.
-
Lab 13 compares exact predicted values; real observations have uncertainty, and a single unreliable observation should not refute a hypothesis on its own.
-
Rebooting first and destroying volatile evidence.
- Stopping at the first hypothesis that fits instead of testing predictions against all observations.
- Calling H4, the changed RTU register map, the cause before a config export confirms it; Lab 13 only shows it is consistent.
- Changing several things at once so that the fix cannot be attributed.
Exercises — chapter 13
EX049 — Predictions first. Objective: practise the method. Inputs: KW-05's seven-minute gap from Chapter 8. Steps: list three hypotheses with their predictions for ping, TCP handshake, Modbus reply, radio counters and other stations. Expected evidence: a table. Rubric: 3 = at least one link hypothesis, one polling/SCADA hypothesis and one security-relevant hypothesis, with distinct predictions so that observations could separate them; 2 = overlapping predictions; 1 = hypotheses only; 0 = none.
EX050 — Add H6. Objective: extend the lab. Inputs: activity 13.6. Steps: implement and run. Expected evidence: code, output and the extra observation. Rubric: 3 = plausible predictions (for example ping true, handshake true, reply timeout or exception 04); recognizes that if H6 predicts exception 02 it is not refuted, and proposes a read-only check such as the RTU's diagnostic page or uptime counter; 2 = runs without reasoning; 1 = attempted; 0 = none.
EX051 — The moment of escalation. Objective: recognize when to stop and preserve. Inputs: the KW-09 case. Steps: write the escalation message to the incident lead and the list of items preserved, with times. Expected evidence: message and list. Rubric: 3 = message states observed facts and the unexplained session, separates inference, asks for a decision on restoration; preserved items include RTU config export and session log, jump-01 and eng-ws-15 logs for the window, the troubleshooting log itself, each with hash; 2 = good but no hashes; 1 = "we've been hacked"; 0 = none.
EX052 — Close the KW-04 path. Objective: fix the bypass found in 13.4. Inputs: the router port-forward and the VPN reachability. Steps: propose the change, its preservation step, its tests and its owner. Expected evidence: change plan. Rubric: 3 = preserve router config first; remove the port-forward; restrict carrier private network from the corporate VPN; negative test from a marked VPN client and positive test of SCADA polling; check all other station routers for the same pattern; owner named; 2 = fix without broad check; 1 = remove only; 0 = none.
Chapter synthesis
Troubleshooting in a critical service is a method, not a hunch: define the symptom, observe read-only, write hypotheses with predictions, eliminate by evidence, preserve, then change one thing and verify. When an observation is unexplained by approved activity, the fault has become a possible incident; stop, preserve and escalate. The log you keep is both the shift handover and the evidence.
Continue studying — chapter 13 of 18
← Chapter 12 · Course chapter index · Chapter 14 →
Continue to the next authored lesson.