Chapter 3 — Critical infrastructure and sector security: specialist evidence

Chapter 3 — Critical infrastructure and sector security: specialist evidence

Lesson map

Lesson 03 published. Map, example, EX011.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["3.1 The specialist's evidential duty"]
  N2["3.2 Evidence sources in a mixed environment"]
  N3["3.3 Time is evidence"]
  N4["3.4 Preservation without harm"]
  N5["3.5 Competing explanations"]
  N6["3.6 Writing findings"]
  N7["3.7 Reporting under obligation"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX011"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["3.1 The specialist's evidential duty"]
  N2["3.2 Evidence sources in a mixed environment"]
  N3["3.3 Time is evidence"]
  N4["3.4 Preservation without harm"]
  N5["3.5 Competing explanations"]
  N6["3.6 Writing findings"]
  N7["3.7 Reporting under obligation"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX011"]

[Learning outcomes and connection]
    |
    v
[3.1 The specialist's evidential duty]
    |
    v
[3.2 Evidence sources in a mixed environment]
    |
    v
[3.3 Time is evidence]
    |
    v
[3.4 Preservation without harm]
    |
    v
[3.5 Competing explanations]
    |
    v
[3.6 Writing findings]
    |
    v
[3.7 Reporting under obligation]
    |
    v
[Worked example] --> [Exercise EX011]

Learning outcomes and connection

After this chapter you should be able to:

1. Identify the evidence sources available in a mixed IT/OT environment and what each can and cannot show.
2. Reconstruct a timeline from sources whose clocks disagree, using anchor events, and state the remaining uncertainty.
3. Plan evidence preservation that does not endanger the physical process.
4. Generate competing explanations and identify the evidence that would discriminate between them.
5. Write findings that separate observation, inference and judgement, at a level of confidence the evidence supports.
6. Meet reporting obligations without overstating what is known.

Chapter 2 graded evidence that controls are in place. This chapter deals with evidence about what happened — the material a specialist brings to an incident review, a regulator, an insurer or a court. Chapter 6 applies the same discipline to planned tests.

3.1 The specialist's evidential duty

When something unexplained happens at an essential service, many people will form a story quickly: the vendor, the operators, the press, the regulator. The specialist's duty is to establish what the evidence supports, at what confidence, and what remains unknown — and to keep those apart under pressure. NIST's incident response recommendations frame response as part of overall risk management, with preparation, detection, analysis and improvement linked to governance (NIST SP 800-61 Rev. 3, 2025). For OT, NIST SP 800-82 Rev. 3 adds that response must protect safety and the process first (NIST, 2023).

The water-sector case from Chapter 2 shows why this matters. In February 2021 an advisory described a compromise of a U.S. water treatment facility's control system through remote-access software (CISA, AA21-042A). In 2023 the city's former manager was reported as saying the event was likely employee error rather than an external intrusion (WUSF, 2023). Whatever the truth, the episode shows how an early public account can harden before the evidence is complete, and how much later analysis depends on what was preserved at the time.

3.2 Evidence sources in a mixed environment

A specialist knows these sources before an incident and fixes the weaknesses that matter most: synchronized time, longer retention for boundary and identity logs, named accounts, and contractual access to supplier records. NIST's control catalogue includes audit-record content, retention and time-stamp requirements for precisely this reason (NIST SP 800-53 Rev. 5).

3.3 Time is evidence

The order of events usually decides the explanation: did the remote session begin before or after the setpoint changed? Each source records time by its own clock, and OT clocks are frequently unsynchronized, drifted or set by hand. Three practices make timelines defensible:

1. Choose a reference clock whose synchronization you can verify (for example a firewall synchronized to a known time source and checked), and express all events in it.
2. Estimate each source's offset from anchor events — a single real-world moment recorded by two sources, such as both ends of the same network session, or a synchronization query passing a firewall. Two independent anchors per source are better than one; disagreement between them reveals drift.
3. State the residual uncertainty for each source — the anchor's own precision, resolution (a handwritten log may have only minutes), and drift over the interval — and check whether any conclusion depends on an ordering closer than that uncertainty.

3.4 Preservation without harm

Evidence preservation in OT must not create the harm it investigates. Principles:

• Safety and process first. The operations lead decides whether a system can be touched. Preservation steps are planned with them, not imposed.
• Collect from the least intrusive source first. Boundary logs, identity logs, historian exports and backups can usually be collected without touching the control network. Controller state should be read with the vendor's supported tools under change control, or not at all until the process is safe.
• Preserve volatile evidence when it is safe. Memory and live connection state on Windows-based HMIs and jump hosts disappear on reboot; capture them if the host can be spared.
• Record chain of custody. Who collected what, when, how, with which hashes — the same discipline as any forensic work. A cryptographic hash such as SHA-256 (FIPS 180-4) shows later that an exported file was not altered.
• Ask suppliers early. Vendor session records and remote-support logs are often overwritten on the supplier's own schedule.

3.5 Competing explanations

A setpoint change with a remote session nearby invites the story "an attacker did it". A specialist lists every explanation that could produce the same evidence and asks what would distinguish them — the method of multiple working hypotheses (Chamberlin, 1965 reprint). For an unexplained dosing change, candidates include an external intruder using a stolen credential, a legitimate vendor engineer making an unrecorded change, an operator mistake on a local HMI, an automated script or recipe, a historian tagging error, and an instrument fault that the operator corrected manually. Each predicts different traces: an intruder's change should come from the remote station after the remote logon; a vendor's planned work should match a ticket and a pattern seen on other sites; an operator error should come from a local station; a recipe should repeat on schedule; a tagging error should disagree with the controller's own record.

3.6 Writing findings

A finding has four parts: the observation with its source; the inference with its assumptions; the confidence in words that map to explicit ranges if your organization uses them; and the consequence for decisions. For example: "The historian records a dosing setpoint change from 1.8 to 2.6 mg/L attributed to station HMI-JH2 at 02:23:03 (firewall time, ±10 s). Station HMI-JH2 is the HMI client on jump host JH-2. The only interactive session on JH-2 at that time was the vendor-svc account, which connected through the VPN from 02:11:40. We assess with moderate confidence that the change was made within the vendor-svc session; we cannot yet say who used that account. The account is shared by four Corvane engineers." This is far more useful to a decision-maker than "the vendor account was compromised".

3.7 Reporting under obligation

Many regimes now require early reporting of significant incidents in stages. NIS2, for example, provides for an early warning, a fuller notification and a final report on fixed timescales (Directive (EU) 2022/2555); NERC CIP and other sector rules have their own clocks. Early reports must be made before the facts are known. The specialist's contribution is to make them accurate about uncertainty: say what has been observed, what is being investigated, and what is not yet known, and avoid attribution that later evidence may reverse. Track every external statement so later reports can correct earlier ones explicitly.

3.8 Worked example: a night at Hollin (synthetic)

At Hollin treatment works, the chlorine dosing setpoint briefly rose from 1.8 to 2.6 mg/L and returned within a minute. The handwritten operator log says the operator noticed an alarm at "02:20" and restored the setpoint. The specialist gathered firewall, jump-host and historian records. Read naively, the historian shows the change at 02:06, before the vendor-svc VPN session began at 02:11 — which would clear the remote session. But the historian server's clock is known to be unsynchronized. Two anchors exist: a TCP session start seen by both the firewall and the jump host, and the historian's own synchronization query passing the firewall.

3.9 Practical activity: timeline across disagreeing clocks

# Lab 3 - reconstructing a timeline from sources whose clocks disagree (offline, standard library only)
# Synthetic: Merrow Water, Hollin treatment works, night of an unexplained chlorine dosing setpoint change.
# Each source records time in its own clock. Anchor events were seen by two sources at the same real moment.
from datetime import datetime, timedelta
T = lambda s: datetime.strptime(s, "%H:%M:%S")
EVENTS = [  # (source, local clock time, description)
    ("firewall",  "02:11:40", "VPN session opened for account vendor-svc"),
    ("firewall",  "02:13:05", "RDP from VPN pool to jump host JH-2 allowed"),
    ("jumphost",  "02:20:31", "interactive logon vendor-svc on JH-2"),
    ("jumphost",  "02:22:58", "HMI client launched on JH-2"),
    ("historian", "02:06:12", "dosing setpoint changed 1.8 -> 2.6 mg/L, source HMI-JH2"),
    ("historian", "02:06:50", "dosing setpoint restored 2.6 -> 1.8 mg/L, source HMI-OPS1"),
    ("ops-log",   "02:20",    "operator notices dosing alarm, restores setpoint (handwritten, minute only)"),
    ("firewall",  "02:31:10", "VPN session closed for vendor-svc"),
]
ANCHORS = [  # the same real moment seen by two sources: (source_a, time_a, source_b, time_b)
    ("firewall", "02:13:07", "jumphost", "02:20:33"),   # TCP session start seen at both ends
    ("firewall", "02:14:30", "historian", "01:57:39"),  # historian's NTP query passing the firewall
]
offset = {"firewall": timedelta(0)}  # firewall is the reference clock (NTP-synchronized, checked)
for a, ta, b, tb in ANCHORS:
    offset[b] = (T(ta) - T(tb)) + offset[a]
offset["ops-log"] = timedelta(0)  # a handwritten log has no known offset, only minute resolution
def show(title, rows):
    print(title)
    for t, src, d in rows: print(f"  {t.strftime('%H:%M:%S')}  {src:9s} {d}")
naive = sorted((T(t) if len(t) > 5 else T(t + ':00'), s, d) for s, t, d in EVENTS)
show("Naive order (raw clocks):", naive)
fixed = sorted(((T(t) if len(t) > 5 else T(t + ':00')) + offset[s], s, d) for s, t, d in EVENTS)
show("Corrected to firewall time:", fixed)
print("offsets:", {k: int(v.total_seconds()) for k, v in offset.items()}, "seconds")

Expected output:

Naive order (raw clocks):
  02:06:12  historian dosing setpoint changed 1.8 -> 2.6 mg/L, source HMI-JH2
  02:06:50  historian dosing setpoint restored 2.6 -> 1.8 mg/L, source HMI-OPS1
  02:11:40  firewall  VPN session opened for account vendor-svc
  02:13:05  firewall  RDP from VPN pool to jump host JH-2 allowed
  02:20:00  ops-log   operator notices dosing alarm, restores setpoint (handwritten, minute only)
  02:20:31  jumphost  interactive logon vendor-svc on JH-2
  02:22:58  jumphost  HMI client launched on JH-2
  02:31:10  firewall  VPN session closed for vendor-svc
Corrected to firewall time:
  02:11:40  firewall  VPN session opened for account vendor-svc
  02:13:05  firewall  RDP from VPN pool to jump host JH-2 allowed
  02:13:05  jumphost  interactive logon vendor-svc on JH-2
  02:15:32  jumphost  HMI client launched on JH-2
  02:20:00  ops-log   operator notices dosing alarm, restores setpoint (handwritten, minute only)
  02:23:03  historian dosing setpoint changed 1.8 -> 2.6 mg/L, source HMI-JH2
  02:23:41  historian dosing setpoint restored 2.6 -> 1.8 mg/L, source HMI-OPS1
  02:31:10  firewall  VPN session closed for vendor-svc
offsets: {'firewall': 0, 'jumphost': -446, 'historian': 1011, 'ops-log': 0} seconds

Interpretation. The jump host runs 446 seconds (7 min 26 s) fast relative to the firewall; the historian runs 1,011 seconds (16 min 51 s) slow. Corrected, the order reverses: the vendor-svc session opens at 02:11:40, the jump-host logon follows the allowed RDP connection within the same second (a consistency check that supports the first anchor), the HMI client starts at 02:15:32, and the setpoint change from station HMI-JH2 occurs at 02:23:03, with a restore from the operators' station 38 seconds later. The naive timeline would have exonerated the remote session; the corrected one places the change inside it. Notice the remaining inconsistency: the handwritten log's "02:20" is three minutes before the change it describes. The likely explanation is that the control-room wall clock is slow, but that is an inference; it should be checked against the wall clock and recorded, not silently "fixed". The finding remains about the session, not a person, and competing explanations (§3.5) — a vendor engineer's legitimate but unrecorded change versus misuse of the shared account — are still open.

3.10 Common errors and limits

• Ordering events across sources by raw timestamps.
• Correcting clocks with a single anchor and never checking it against a second.
• Treating a historian "source" tag as a person.
• Touching controllers to collect evidence without the operations lead's decision.
• Letting the first plausible story define what evidence is collected.
• Writing "compromised" or "attack" in early reports when the evidence supports only "unexplained change during a remote session".
• Forgetting that supplier records expire.

The limit is that evidence can narrow explanations but rarely eliminates all but one. A specialist reports which explanations remain and what would separate them; the decision to act — for example suspending a vendor's access — can and often should be taken before attribution is settled.

Exercises — chapter 3

EX011 — Robustness of the order. Objective: quantify how much a timeline conclusion depends on one anchor. Inputs: Lab 3. Constraints: consider errors only in the historian anchor. Steps: (1) compute how many seconds earlier the corrected setpoint change would have to be for it to precede the HMI client launch on JH-2; (2) change the historian anchor's firewall time by one minute (02:14:30 → 02:15:30), rerun, and report whether the order changes; (3) state what independent evidence would confirm the historian offset. Expected evidence: calculation, rerun output and paragraph. Rubric: 3 = margin computed (451 s), rerun shows the order is unchanged, a second independent anchor proposed; 2 = one part missing; 1 = not run; 0 = none.

EX012 — Preservation plan. Objective: plan safe evidence collection. Inputs: the Hollin event. Constraints: the plant must keep producing water; the operations lead has authority over control systems. Steps: list sources in collection order with method, who approves, volatility and risk to process; include chain of custody and supplier requests. Expected evidence: table and a one-paragraph rationale. Rubric: 3 = least-intrusive first, JH-2 volatile capture before any reboot, controller reads under operations approval with vendor tools, hashes and custody, Corvane session records requested immediately; 2 = order partly wrong; 1 = endangers process; 0 = none.

EX013 — Write the finding. Objective: separate observation, inference, confidence and consequence. Inputs: Lab 3 output. Constraints: 200 words; no attribution to a person. Steps: write the finding for the board and a one-sentence version for the regulator's early notification. Expected evidence: both texts. Rubric: 3 = observations sourced with clock basis and uncertainty, inference explicit, confidence stated, consequence for decisions (for example suspend shared account) clear, regulator sentence accurate about unknowns; 2 = one element merged; 1 = asserts compromise as fact; 0 = none.

EX014 — Discriminating evidence. Objective: apply multiple working hypotheses. Inputs: the six candidate explanations in §3.5. Constraints: for each, one prediction that the others do not share. Steps: build a table of explanation, distinguishing prediction, evidence source and current status (supported, contradicted, open). Expected evidence: table. Rubric: 3 = predictions genuinely discriminating, statuses consistent with Lab 3 (operator error at a local station contradicted for the change itself; vendor-legitimate versus misuse open), next evidence identified; 2 = some predictions shared; 1 = list only; 0 = none.

EX015 — Staged reporting. Objective: draft external statements that survive later evidence. Inputs: the Hollin event; assume Merrow Water is subject to a staged incident-reporting regime similar to NIS2 Article 23. Constraints: do not interpret real law for a real entity; use placeholders for deadlines. Steps: draft the early warning, the notification and the outline of the final report, showing how each corrects or extends the previous one. Expected evidence: three short texts. Rubric: 3 = early warning factual and uncertainty-explicit, notification adds the corrected timeline and actions, final report outline includes root cause, corrections and lessons; 2 = early text over-claims; 1 = attribution asserted; 0 = none.

Chapter synthesis

Specialist evidence is the material decisions and accountability rest on. Know your sources before an incident and fix the weaknesses that matter — time, retention, named accounts and supplier access. Treat time as evidence: choose a reference clock, estimate offsets from anchors, and state residual uncertainty. Preserve without harming the process, least intrusive first, with chain of custody. Generate competing explanations and seek evidence that discriminates. Write findings that keep observation, inference, confidence and consequence apart, and report early in a way later evidence will extend rather than contradict.

Continue studying — chapter 03 of 12

← Chapter 02 · Course chapter index · Chapter 04 →

Continue to the next authored lesson.