Chapter 6 — Critical infrastructure and sector security: operational workflow
Lesson map
Lesson 06 published. Map, example, EX026.
flowchart TD
N0["Learning outcomes and connection"]
N1["6.1 Roles"]
N2["6.2 RACI"]
N3["6.3 Change governance across IT and OT"]
N4["6.4 Escalation"]
N5["6.6 Practical activity"]
N6["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX026"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["6.1 Roles"]
N2["6.2 RACI"]
N3["6.3 Change governance across IT and OT"]
N4["6.4 Escalation"]
N5["6.6 Practical activity"]
N6["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX026"]
[Learning outcomes and connection]
|
v
[6.1 Roles]
|
v
[6.2 RACI]
|
v
[6.3 Change governance across IT and OT]
|
v
[6.4 Escalation]
|
v
[6.6 Practical activity]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX026]
Learning outcomes and connection
After this chapter you should be able to:
- define the roles a critical-infrastructure security programme needs and how they relate to operations;
- build and check a RACI matrix so that every key activity has exactly one accountable person and a clear lead;
- run change governance across IT and OT without either blocking operations or bypassing safety;
- design an escalation path from a shift operator to the executive that works at 03:00 on a Sunday.
Chapter 5 gave you yardsticks. This chapter makes the programme run, day to day.
6.1 Roles
At Kestrel Water's size, the roles below may be shared by a handful of people. What matters is that each is assigned:
- Executive sponsor (chief operating officer) — owns the programme, accepts high risks, resolves conflicts between departments.
- Service owners (operations manager for treatment and stations; customer-services manager for billing) — own risks and decisions for their services.
- Coordinator — runs the cycle: register, records, exercises, reporting, supplier reviews.
- Security lead — technical security advice and monitoring liaison with KVROC.
- Control-systems engineer — configuration and change on operational systems.
- Communications lead — public and media messages.
- Incident lead (rotating, on call) — commands the response when an incident is declared.
- Suppliers — do work under authorization; never own decisions about your service.
NIST's incident response guidance emphasises defining roles and responsibilities, including those outside the security team, before incidents (NIST SP 800-61 Rev. 3).
6.2 RACI
A RACI matrix lists activities against people with one letter each: Responsible (does the work), Accountable (answers for the outcome and signs off — exactly one per activity), Consulted (two-way input before the decision), Informed (told after). Two rules catch most defects:
- Exactly one A per activity. Zero means nobody answers; two means each assumes the other will.
- At least one R, and where several, one named lead.
Add two coordinator's checks: nobody should be accountable for approving their own access or work; and nobody should carry so many R and A entries in an incident that they become a bottleneck.
6.3 Change governance across IT and OT
Changes to operational systems can affect the physical process. A workable change process for a small utility has three classes:
- Standard — pre-approved, low-risk, repeatable (replacing a failed operator keyboard): log it.
- Normal — assessed and approved by the service owner, with a test or rollback plan and a window agreed with operations.
- Emergency — needed now to protect people or the service; approved verbally by the incident lead or service owner, recorded within 24 hours and reviewed afterwards.
The coordinator's interest is that security-relevant changes (new remote access, firewall rule changes, new suppliers, software on control-room computers) cannot slip into "standard", and that emergency changes made during incidents are reviewed rather than becoming permanent.
6.4 Escalation
An escalation path answers, for each severity: who is called, by whom, how, within what time, and what happens if they do not answer. Write it as a table with deputies, test it (Chapter 11), and put it where night-shift operators can find it without a computer — a printed card in the control room — because the incident might have taken the computers.
Kestrel Water's card lists the incident lead, the operations manager and the security lead with a named deputy for each, plus mobile numbers that do not depend on the corporate network.
6.5 Worked example: checking the incident RACI (synthetic)
The coordinator drafts Kestrel Water's incident RACI and runs Lab 6, which applies the rules of 6.2.
Lab 6 (save as lab06.py, run with python3 lab06.py):
# Lab 6 - RACI matrix check for incident coordination (offline, standard library only; synthetic)
PEOPLE = ["Coordinator", "Ops manager", "Security lead", "Comms", "Vendor", "CEO"]
# activity -> one letter per person (R responsible, A accountable, C consulted, I informed, - none)
RACI = {
"Declare cyber incident": "R A C I - I",
"Decide to isolate stations": "C A R I C I",
"Switch stations to manual operation":"I A - - - I",
"Notify regulator": "R - C C - A",
"Brief hospital (critical customer)":"R C I R - A",
"Approve vendor remote session": "C A A - R -",
"Public statement": "C C C R - -",
}
issues = []
load = {p: 0 for p in PEOPLE}
for act, codes in RACI.items():
c = codes.split()
assert len(c) == len(PEOPLE), act
a = [p for p, x in zip(PEOPLE, c) if x == "A"]
r = [p for p, x in zip(PEOPLE, c) if x == "R"]
for p, x in zip(PEOPLE, c):
if x in "RA": load[p] += 1
if len(a) != 1: issues.append(f"{act}: {len(a)} accountable ({', '.join(a) or 'none'})")
if not r: issues.append(f"{act}: nobody responsible")
if len(r) > 1: issues.append(f"{act}: {len(r)} responsible ({', '.join(r)}) - name a lead")
if act.startswith("Approve") and c[PEOPLE.index("Vendor")] in "RA":
issues.append(f"{act}: vendor is responsible for approving its own access")
for i in issues: print("ISSUE", i)
print("R/A load:", ", ".join(f"{p}={n}" for p, n in load.items()))
print(f"{len(issues)} issues in {len(RACI)} activities")
Expected output (Python 3.13.5):
ISSUE Switch stations to manual operation: nobody responsible
ISSUE Brief hospital (critical customer): 2 responsible (Coordinator, Comms) - name a lead
ISSUE Approve vendor remote session: 2 accountable (Ops manager, Security lead)
ISSUE Approve vendor remote session: vendor is responsible for approving its own access
ISSUE Public statement: 0 accountable (none)
R/A load: Coordinator=3, Ops manager=4, Security lead=2, Comms=2, Vendor=1, CEO=2
5 issues in 7 activities
Five defects in seven activities:
- Switch stations to manual operation has an accountable owner (operations manager) but nobody responsible — in practice, shift operators and on-call technicians do it, and they are missing from the matrix. The fix is to add the shift supervisor as R.
- Brief hospital has two responsibles. The coordinator and communications might each think the other called. Name the coordinator as lead (it is an operational liaison) with communications consulted on wording.
- Approve vendor remote session has two accountables and lists the vendor as responsible. The operations manager should be the single A (owns the stations); the security lead is C; the vendor requests access and is I about the decision — a party must never approve its own access.
- Public statement has no accountable person. The chief executive should be A, communications R.
The load line shows the operations manager carrying four R/A entries — during an incident, the same person is running the plant. The coordinator proposes a deputy for the incident-related entries. The lab checks structure only; whether the right people were chosen is a judgement for the executive sponsor.
6.6 Practical activity
Write a RACI for "onboard a new supplier with remote access" using at least five roles, then check it against the four rules by hand or by editing Lab 6.
Include the supplier as a role and make sure it never approves its own access, which Lab 6 checks explicitly. Name who is accountable for creating the account, who checks that the contract has the security clauses, and who is informed when the account is enabled. Then count each person's R and A entries: if one name appears on nearly every line, onboarding will stall whenever that person is on leave. Show the result to the operations manager and ask whether anyone is missing.
6.7 Common errors and limits
- Committees as accountable ("the security board").
- RACIs written for normal days only, not incidents.
- Emergency changes never reviewed.
- Escalation lists without deputies, or only stored on the network.
-
A RACI is a social contract: publish it, walk through it with each person, and update it after reorganisations.
-
Leaving the people who actually do the work off the matrix, as Lab 6 found with shift operators for manual operation.
- Allowing a party to approve its own access, such as a supplier marked responsible for approving its remote session.
- Giving one person so many R and A entries that the incident depends on them alone.
Exercises — chapter 6
EX026 — Fix the RACI. Objective: correct defects. Inputs: Lab 6 matrix. Steps: propose corrected rows for the five issues and rerun the check (by hand or code). Expected evidence: corrected matrix and clean result. Rubric: 3 = zero issues, vendor never A or R for approval, a shift role added for manual operation, a single lead for hospital briefing, CEO accountable for public statement; 2 = one issue remains; 1 = partial; 0 = none.
EX027 — Classify changes. Objective: apply change classes. Inputs: (a) new firewall rule allowing a supplier VPN; (b) replacing a failed UPS battery in a station cabinet; (c) disabling a compromised account during an incident; (d) installing a monitoring agent on the historian. Steps: classify and justify. Expected evidence: four classes. Rubric: 3 = (a) normal (security-relevant), (b) standard if pre-approved procedure exists, (c) emergency with post-review, (d) normal with supplier compatibility check; 2 = three right; 1 = all standard; 0 = none.
EX028 — Escalation card. Objective: design a usable escalation path. Inputs: roles in 6.1. Steps: write the printed card for night-shift operators for a suspected cyber event. Expected evidence: card content. Rubric: 3 = observable triggers, first call and deputy with phone numbers placeholders, time limits, what to do meanwhile (safety first, do not switch off computers without instruction, note times), when to call the incident lead directly; 2 = no deputies; 1 = "call IT"; 0 = none.
EX029 — Bottleneck analysis. Objective: use the load line. Inputs: Lab 6 load output. Steps: propose how to reduce the operations manager's incident load without weakening safety authority. Expected evidence: proposal. Rubric: 3 = keeps safety decisions with operations but delegates liaison and vendor approval to a named deputy or the incident lead, with the operations manager consulted; 2 = delegation without safety caveat; 1 = moves safety decisions to IT; 0 = none.
EX030 — Emergency change review. Objective: close emergency changes. Inputs: during an incident, an engineer added a firewall rule blocking all supplier access and created a temporary local admin account. Steps: describe the post-incident review. Expected evidence: review steps. Rubric: 3 = list emergency changes from the record, decide keep/modify/revert for each with owner, remove temporary account, restore supplier access only through the normal process, update documentation; 2 = some steps; 1 = "leave it"; 0 = none.
Chapter synthesis
A programme runs on assigned roles, a RACI with one accountable per activity and a named lead, change governance that keeps security-relevant changes visible, and an escalation path that works offline at night. Check the RACI for structure, ownership of approvals and incident load; review every emergency change after the event.
Continue studying — chapter 06 of 12
← Chapter 05 · Course chapter index · Chapter 07 →
Continue to the next authored lesson.