Chapter 29 — Critical infrastructure and sector security: operations

Lesson map

Lesson 29 published. Map, example, EX057.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["29.1 What security operations do"]
  N2["29.2 Incident response in OT"]
  N3["29.3 Measuring operations"]
  N4["29.5 A playbook step synthetic"]
  N5["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX057"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["29.1 What security operations do"]
  N2["29.2 Incident response in OT"]
  N3["29.3 Measuring operations"]
  N4["29.5 A playbook step synthetic"]
  N5["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX057"]

[Learning outcomes and connection]
    |
    v
[29.1 What security operations do]
    |
    v
[29.2 Incident response in OT]
    |
    v
[29.3 Measuring operations]
    |
    v
[29.5 A playbook step synthetic]
    |
    v
[Chapter synthesis]
    |
    v
[Worked example] --> [Exercise EX057]

Learning outcomes and connection

After this chapter you can (1) describe security operations for a critical-service operator — monitoring, triage, escalation, change and vulnerability management; (2) adapt incident response to operational technology, where keeping the process safe comes first; (3) measure operational performance with time-to-detect and time-to-recover using medians and percentiles; and (4) write a playbook step that respects both security and operations. It builds on Chapters 12, 15, 24, 27 and 28 and prepares Laboratory II (Chapter 30).

29.1 What security operations do

Day to day, a security operations function monitors (logs, network traffic, alerts), triages (is this real? how bad?), escalates (to whom, how fast), coordinates change (no surprise changes on OT systems), and manages vulnerabilities (Chapters 5, 9, 11). For a critical-service operator, the defining feature is the partnership with operations: control-room staff, engineers and clinicians know what normal looks like for the process, and they own the decisions that affect it. Security operations that do not talk to the control room will misread alerts and, worse, take actions that hurt the service.

29.2 Incident response in OT

NIST's incident-response recommendations, revised in 2025, place response within overall cybersecurity risk management and the CSF 2.0 functions (SP 800-61 Rev. 3). In OT the order of priorities shifts: first keep the process safe (which may mean moving to manual or local control), then contain (isolate affected IT or OT segments without cutting what the process needs), then investigate and recover. Disconnecting a compromised server is reflexive in IT; in OT the operator must first know what that server does — pulling the historian is harmless, pulling the only SCADA server may blind operators during a storm. Evidence must also be preserved: images of engineering workstations, controller configurations, network captures and the operator log. Playbooks are agreed in advance with operations, safety and, where personal data may be involved, data protection (Chapter 14).

29.3 Measuring operations

Common measures are time to detect (TTD: first malicious or faulty activity to detection) and time to recover (TTR: detection to restored service). Their distributions are skewed (Chapter 12): most incidents are handled quickly; a few take days. Report median, 90th percentile and sample size, and inspect the outliers individually — the long tail is where lessons live. Beware of incentives: a TTD target can encourage logging the "detection" time generously.

Lab 12's 15 records show why: mean 27.2 hours, median 4.

Inspect the 240-hour detection individually; it was a slow credential misuse.

29.4 Worked example and Lab 12 (synthetic)

This case is synthetic. CRP members pooled 15 sanitised incident records from the last two years.

# Lab 12 - time to detect and recover: mean, median and percentiles from synthetic tickets (offline)
import statistics as st
TTD_H = [0.5, 1, 1, 2, 2, 3, 3, 4, 6, 8, 12, 20, 36, 70, 240]   # hours from first malicious activity to detection
TTR_H = [2, 3, 3, 4, 5, 5, 6, 8, 8, 10, 12, 16, 24, 48, 96]
for name, xs in (("time to detect", TTD_H), ("time to recover", TTR_H)):
    q = st.quantiles(xs, n=10, method="inclusive")
    print(f"{name}: n={len(xs)} mean={st.mean(xs):.1f} median={st.median(xs):.1f} p90={q[8]:.1f} max={max(xs)}")
print("mean TTD without the 240 h outlier:", round(st.mean(TTD_H[:-1]), 1))

Expected output:

time to detect: n=15 mean=27.2 median=4.0 p90=56.4 max=240
time to recover: n=15 mean=16.7 median=8.0 p90=38.4 max=96
mean TTD without the 240 h outlier: 12.0

The mean time to detect (27.2 h) describes no real incident: half were detected within 4 hours, and one took ten days. Removing that single record cuts the mean by more than half. The useful report says: "median 4 h, 90th percentile about 56 h, n = 15; the 240-hour case was a slow credential misuse found during an access review" — and then asks why monitoring missed it. Percentiles on 15 records are rough; the method of computing them should be stated (here, the "inclusive" method of Python's statistics.quantiles).

29.5 A playbook step (synthetic)

CPN playbook, step 3 — suspected compromise of an engineering workstation: (1) Security analyst informs the control-room supervisor; supervisor confirms no switching operation depends on the workstation now. (2) Supervisor authorises isolation; analyst isolates the workstation's network port, leaving it powered (to preserve memory). (3) Engineer checks controller configurations the workstation could have changed, comparing with the last known-good backup. (4) Analyst records times, decisions and authorisers in the incident log. Each step names who decides and why the order matters.

If a CPN engineer were mid-way through switching load around Substation North from that workstation, pulling its network port would leave the operation half done, with the control room unsure which breakers had moved.

29.6 Common errors and limits

• Security teams acting on OT without operations' authorisation.
• Pulling power or wiping systems before evidence is preserved.
• Reporting mean times for skewed data.
• Playbooks that exist only on the systems the incident has disabled.

The CPN playbook avoids the first two by having the supervisor authorise isolation and leaving the workstation powered. Lab 12 shows the third. The fourth is why CRP members keep printed copies of playbooks in each control room, since an incident may disable the document system that holds them.

Exercises — chapter 29

EX057 — Report the tail. Objective: summarise skewed data honestly. Inputs: Lab 12. Steps: write a 100-word summary for the CRP board of TTD and TTR, including sample size, one limitation and one question about the longest case. Expected evidence: the summary. Rubric: 3 = medians and p90 with n, mean's distortion noted, limitation (small, sanitised, self-reported detection times), pointed question; 2 = numbers without caveat; 1 = means only; 0 = none.

EX058 — Safe containment. Objective: order actions in OT response. Inputs: ransomware detected on CW's historian server in the control zone; SCADA servers appear unaffected. Steps: write the first five actions in order with who decides each. Expected evidence: ordered list. Rubric: 3 = confirm process state with operations; prepare manual/local operation; isolate historian from control zone with operations' approval; preserve evidence; check SCADA and controllers against known-good, with roles named; 2 = sensible but unordered or roles missing; 1 = IT-style "shut everything down"; 0 = none.

Chapter synthesis

Security operations in critical services work hand in hand with operations: keep the process safe first, contain without blinding operators, preserve evidence and recover. Measure with medians, percentiles and sample sizes, and investigate the outliers that averages hide.