Chapter 20 — Critical infrastructure and sector security: troubleshooting
Lesson map
Lesson 20 published. Map, example, EX058.
flowchart TD
N0["Learning outcomes and connection"]
N1["20.1 A method"]
N2["20.2 Preserve before you change"]
N3["20.3 When a fault becomes an incident"]
N4["20.4 Missing data: four common causes"]
N5["20.6 Practical activity: discriminate the causes"]
N6["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX058"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["20.1 A method"]
N2["20.2 Preserve before you change"]
N3["20.3 When a fault becomes an incident"]
N4["20.4 Missing data: four common causes"]
N5["20.6 Practical activity: discriminate the causes"]
N6["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX058"]
[Learning outcomes and connection]
|
v
[20.1 A method]
|
v
[20.2 Preserve before you change]
|
v
[20.3 When a fault becomes an incident]
|
v
[20.4 Missing data: four common causes]
|
v
[20.6 Practical activity: discriminate the causes]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX058]
Learning outcomes and connection
After this chapter you should be able to:
- follow a disciplined troubleshooting method that works whether a fault turns out to be technical, human or malicious;
- preserve evidence before changing things, and know when a fault must become an incident;
- distinguish common causes of "missing data" — loss, delay, clock error, filtering — using the data themselves;
- write a short problem record that the next person can use.
Most of an implementer's working life is spent on faults. In critical services some faults are attacks and some attacks look like faults; the method must work for both without slowing down ordinary repairs.
20.1 A method
- Define the symptom precisely: what is wrong, since when, for whom, measured how. "PS-07 and PS-09 show gaps on the dashboard since about 22:50" — not "telemetry is broken".
- Check impact and safety first: does anyone need to act on the physical process now? Tell Water operations what they can and cannot rely on.
- Preserve before you change: copy the relevant logs, configurations and outputs, with times, before restarting or reconfiguring anything (see "Preserve before you change" below).
- List hypotheses, including at least one non-malicious and one malicious explanation, and what evidence would distinguish them.
- Test the cheapest discriminating hypothesis first, using reads before writes.
- Change one thing at a time, under the appropriate change type (Chapter 14), recording each change.
- Verify the fix against the original symptom, and watch for recurrence.
- Record cause, evidence, actions and open items.
20.2 Preserve before you change
Restarting a service clears its memory, rotates logs, and may destroy the only evidence of what happened. NIST's incident-response recommendations stress preserving evidence and coordinating response with the organization's risk management (NIST SP 800-61 Rev. 3). For an implementer, preservation is usually quick: copy the service's logs and configuration, export the relevant flow and alert records, note the running processes and network connections, take hashes of what you copied, and write down the time. If the fault affects the physical process, restoring service may legitimately come first — but say so and record what was lost.
20.3 When a fault becomes an incident
Escalate to the incident process when any of these is true: evidence of unauthorized access or change (an unapproved configuration difference, a leaver's account used — Chapter 7); a security control has failed in a way that could be exploited; data may have gone where they should not (Chapter 12); or the cause cannot be found and a malicious cause is plausible. Escalating is not an accusation. It brings in people with different authority and duties, including the operator's reporting obligations (Chapter 10).
c.mendes's login on 14 September is the clearest Fallowmere trigger.
20.4 Missing data: four common causes
Missing telemetry is LANTERN's commonest symptom. The data themselves usually discriminate between causes:
- Lost — sequence numbers have gaps (Chapter 15): messages were never received.
- Delayed — sequence complete, but receive time far behind station time, often after a carrier problem.
- Clock error — sequence complete and arrival prompt, but station timestamps offset by a constant amount; data are filed in the wrong time window, so a dashboard querying by station time shows a gap.
- Filtered — messages received but rejected or quarantined (rising reject counts in the relay log).
20.5 Worked example: two gaps, two causes (synthetic)
Operators report dashboard gaps for PS-07 and PS-09. Lab 20 examines twelve sequence numbers from each station. PS-07 is missing sequence numbers 5–8 with prompt arrival for the rest: data were lost. PS-09 has every sequence number and arrives promptly by the relay's clock, but station timestamps are about 422 seconds behind: its clock is wrong, and the dashboard window misses all its points. The two symptoms looked identical on the dashboard; the causes and the fixes are completely different.
For PS-09, the next question is why the clock is wrong: has the station lost its NTP path to the relay (a conduit change? Chapter 2), or has someone changed the station's time configuration? The first is a fault; the second may be a security event. A constant offset of about seven minutes on every message is unusual for drift — drift is gradual. Preserve the relay's NTP logs and the conduit configuration before anyone resets the clock.
For PS-07, the gap of four consecutive messages could be a carrier drop, a station restart, or relay rejects. Check the relay's reject log for those times before calling BayTel: if there were MAC failures at PS-07 then, the cause is in key handling (Chapter 18), not the carrier.
20.6 Practical activity: discriminate the causes
Lab 20 (save as lab20.py, run with python3 lab20.py):
# Lab 20 - troubleshooting "missing telemetry": sequence gaps versus clock skew (offline, standard library only)
# Synthetic historian extract: station, sequence number, station timestamp (s), relay receive time (s).
rows = {
"PS-04": [(s, 1000 + 10*s, 1000 + 10*s + 1) for s in range(1, 13)],
"PS-07": [(s, 1000 + 10*s, 1000 + 10*s + 1) for s in range(1, 13) if s not in (5, 6, 7, 8)],
"PS-09": [(s, 1000 + 10*s - 420, 1000 + 10*s + 2) for s in range(1, 13)],
}
for st, rs in rows.items():
seqs = [r[0] for r in rs]
missing = sorted(set(range(seqs[0], seqs[-1] + 1)) - set(seqs))
offsets = [recv - ts for _, ts, recv in rs]
avg = sum(offsets) / len(offsets)
dx = "data lost in transit or at source" if missing else ("clock skew: station clock behind by ~%d s; data present but filed in the wrong window" % round(avg) if abs(avg) > 60 else "healthy")
print(f"{st}: received {len(rs)}/{seqs[-1]-seqs[0]+1}, missing seq {missing or '-'}, mean recv-ts offset {avg:+.0f} s -> {dx}")
# A dashboard that queries by station timestamp for window [1050, 1130) would show:
for st, rs in rows.items():
print(st, "points in dashboard window:", sum(1 for _, ts, _ in rs if 1050 <= ts < 1130))
Expected output (Python 3.13.5):
PS-04: received 12/12, missing seq -, mean recv-ts offset +1 s -> healthy
PS-07: received 8/12, missing seq [5, 6, 7, 8], mean recv-ts offset +1 s -> data lost in transit or at source
PS-09: received 12/12, missing seq -, mean recv-ts offset +422 s -> clock skew: station clock behind by ~422 s; data present but filed in the wrong window
PS-04 points in dashboard window: 8
PS-07 points in dashboard window: 4
PS-09 points in dashboard window: 0
Look at the two stations' columns side by side. PS-07's sequence numbers jump from 4 to 9, so four messages never arrived. PS-09's sequence numbers are complete, but its station timestamps lag the relay's arrival times by about 422 seconds on every message. The first is loss; the second is time. Then write the next check for each: the relay's reject log and BayTel's status for the PS-07 gap, and the NTP path and time configuration for PS-09, preserved before anyone resets the clock.
20.7 Common errors and limits
- Restarting first and investigating later.
- One hypothesis, usually the most familiar.
- Several changes at once, so nobody knows which fixed it — or which broke something else.
- Treating every fault as an attack, or none.
- "Fixing" a clock or a configuration before recording what it was.
- Closing the ticket without a record of cause and evidence.
Resetting PS-09's clock first would have destroyed the evidence for the 422-second question.
Calling BayTel for PS-07's gap before checking the relay's reject log would be the second error: one hypothesis, the most familiar one, tested first and possibly only.
Exercises — chapter 20
EX058 — Hypotheses and discriminators. Objective: reason before acting. Inputs: symptom: "relay reject count for PS-04 rose from 0 to 300 per hour at 09:00; dashboard shows no PS-04 data since." Constraints: four hypotheses, at least one malicious. Steps: for each, give the evidence that would support or rule it out, and the cheapest check. Expected evidence: table. Rubric: 3 = for example key rotated at station but not relay (MAC failures in reject reasons, change calendar), station clock jump (window rejects), replay state poisoned (replay rejects after a high sequence — Chapter 29), forged traffic (MAC failures from an unexpected source address); checks are reads; 2 = three; 1 = one; 0 = none.
EX059 — Preservation list. Objective: preserve evidence quickly. Inputs: PS-09 clock offset case. Constraints: ten minutes maximum; no changes to systems. Steps: list what you would copy or record, in order, and how you would show later that the copies were not altered. Expected evidence: list. Rubric: 3 = relay NTP and relay logs, conduit/firewall configuration and recent change log, historian extract, Lab-20-style analysis output, time of collection, hashes recorded in the ticket and copies stored off-host; 2 = missing hashing or times; 1 = "take a screenshot"; 0 = none.
EX060 — Problem record. Objective: write a usable record. Inputs: Chapter 20 ("Worked example: two gaps, two causes") and assume the PS-09 cause was a firewall change that blocked udp/123 from relay to station two days earlier. Constraints: at most 250 words; observed / inferred / done / open labels. Steps: write the record. Expected evidence: the record. Rubric: 3 = precise symptom, evidence (offset ~422 s, complete sequence), cause with the change identified, fix under change control, verification, and open items (why the change removed NTP; why no alert fired on clock offset — add one); 2 = one element missing; 1 = narrative only; 0 = none.
Chapter synthesis
Troubleshoot with a method: precise symptom, safety and impact first, preserve before changing, several hypotheses including a malicious one, cheapest discriminating check first, one change at a time, verify and record. Escalate to an incident when evidence suggests unauthorized activity or exposure. For missing data, the data themselves — sequence, arrival, timestamps, reject counts — usually tell loss, delay, clock error and filtering apart.
Continue studying — chapter 20 of 30
← Chapter 19 · Course chapter index · Chapter 21 →
Continue to the next authored lesson.