◇ OpenCyber Academy
Technologist · Chapter 23 · English

Chapter 23 — Critical infrastructure and sector security: evaluation

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 23 — Critical infrastructure and sector security: evaluation

Lesson map

Lesson 23 published. Map, example, EX067.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["23.1 What to test"]
  N2["23.2 Rules of engagement"]
  N3["23.3 Claim rules"]
  N4["23.4 From results to work"]
  N5["23.6 Practical activity: evaluate controls"]
  N6["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX067"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["23.1 What to test"]
  N2["23.2 Rules of engagement"]
  N3["23.3 Claim rules"]
  N4["23.4 From results to work"]
  N5["23.6 Practical activity: evaluate controls"]
  N6["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX067"]

[Learning outcomes and connection]
    |
    v
[23.1 What to test]
    |
    v
[23.2 Rules of engagement]
    |
    v
[23.3 Claim rules]
    |
    v
[23.4 From results to work]
    |
    v
[23.6 Practical activity: evaluate controls]
    |
    v
[Chapter synthesis]
    |
    v
[Worked example] --> [Exercise EX067]

Learning outcomes and connection

After this chapter you should be able to:

Chapters 18 and 19 implemented controls and rules. This chapter asks whether they work. NIST's technical guide to security testing describes planning, conducting and reporting tests, including the need for authorization and rules of engagement (NIST SP 800-115). In critical services the emphasis is on tests that are safe for the physical process.

23.1 What to test

Test each control against the evidence its card promises (Chapter 18). For detective controls, inject a safe test event that should produce the alert: a deliberately failed login burst from a test address; a marked test flow from a test workstation to a test port on a spare PLC; a paused test heartbeat. For preventive controls, attempt the prevented action in a controlled way: a message with a wrong MAC; a connection from a non-permitted zone. For recovery controls, run a timed restore (Chapter 16).

23.2 Rules of engagement

Before any test, write and get signed rules of engagement: purpose; systems in scope (by identifier) and explicitly out of scope; permitted actions; time window; who is on site or on call; communication plan (who is told before, during and after); stop conditions (any unexpected process behaviour stops the test at once); how test events are marked so responders and operators recognize them; and how results are stored. In the station zone, prefer tests on spare or laboratory equipment, or on the relay side of the conduit, and never send test traffic to a live controller without operations' explicit agreement for that controller and time.

23.3 Claim rules

Decide before testing what result will count as "effective". A claim rule might be: the 95 % Wilson lower bound of the detection rate is at least 0.70 and the median time to alert meets the target. Fixing the rule in advance stops the temptation to choose a flattering interpretation afterwards. Chapter 9's lesson applies: small numbers of tests give wide intervals, so "8 of 8 detected" may still not meet the rule.

Rule C's 8 of 8 is the case: a perfect record whose lower bound, 0.68, misses 0.70.

23.4 From results to work

Each control that is "not yet shown" needs either improvement (fix the rule, the source, the configuration) or more tests (if the issue is only sample size). Record the next test date. Each control "shown effective" is re-tested on a schedule, because controls degrade.

Lab 23 sorts its four controls this way. Rule C needs only more tests: one further success would just reach 0.70 (0.701), and a second would give a margin (0.722). Rule B needs improvement first, by exporting the missing station switch's flows to the SIEM, then a retest. FIM needs both: more frequent checks on critical files, then more tests. Backup restore needs the decryption key added to the runbook, then new drills.

23.5 Worked example: a quarter's control tests (synthetic)

Lab 23 applies the claim rule to six controls. Two are shown effective: Rule A detected 12 of 12 brute-force tests in a median of 2 minutes (lower bound 0.76), and the relay rejected all 40 MAC-failure tests (lower bound 0.91). Four are not yet shown, for different reasons:

23.6 Practical activity: evaluate controls

Lab 23 (save as lab23.py, run with python3 lab23.py):

# Lab 23 - evaluating whether the service's controls work, not only whether they exist (offline, standard library only)
# Claim rule used here: 'shown effective' needs the 95% Wilson lower bound >= 0.70 and the time target met.
# Synthetic results of a quarter of control tests at Fallowmere. Each test injects a safe, authorized test event.
import math
tests = {  # control: (test events injected, detected/blocked as specified, median minutes to alert)
 "brute-force alert (rule A)": (12, 12, 2),
 "corp->station Modbus alert (rule B)": (10, 7, 4),
 "station-silent alert (rule C)": (8, 8, 16),
 "MAC rejection at relay": (40, 40, None),
 "FIM unapproved-change alert": (6, 4, 1440),
 "backup restore within 4 h": (10, 9, None),
}
def wilson_low(k, n, z=1.96):
    p = k / n; d = 1 + z*z/n; c = (p + z*z/(2*n)) / d; h = z*math.sqrt(p*(1-p)/n + z*z/(4*n*n)) / d
    return c - h
target = {"FIM unapproved-change alert": 60, "station-silent alert (rule C)": 20}
for name, (n, k, mins) in tests.items():
    lo = wilson_low(k, n)
    late = name in target and mins is not None and mins > target[name]
    verdict = "shown effective" if lo >= 0.70 and not late else "not yet shown"
    print(f"{name:<36} {k:>2}/{n:<2} low95={lo:.2f} median {'-' if mins is None else mins:>4} min  {verdict}{' (too slow)' if late else ''}")

Expected output (Python 3.13.5):

brute-force alert (rule A)           12/12 low95=0.76 median    2 min  shown effective
corp->station Modbus alert (rule B)   7/10 low95=0.40 median    4 min  not yet shown
station-silent alert (rule C)         8/8  low95=0.68 median   16 min  not yet shown
MAC rejection at relay               40/40 low95=0.91 median    - min  shown effective
FIM unapproved-change alert           4/6  low95=0.30 median 1440 min  not yet shown (too slow)
backup restore within 4 h             9/10 low95=0.60 median    - min  not yet shown

The two effective controls are Rule A, 12 of 12 with lower bound 0.76, and the relay's MAC check, 40 of 40 with lower bound 0.91. For each of the four that are not yet shown, write one line: the reason (Rule B, a source gap; Rule C, sample size; FIM, daily checks and too few tests; restore, a runbook gap), the next action and the next test date. Do not round Rule C's 0.68 up to 0.70.

23.7 Common errors and limits

Tests show how controls respond to the test events you designed. Real adversaries and real faults may differ; vary tests over time and include scenarios from incidents elsewhere.

Exercises — chapter 23

EX067 — Rules of engagement. Objective: authorize a safe test. Inputs: a test of Rule B using a test workstation and the spare Arlen 300 PLC in FSTS's lab, plus one test flow to PS-04's PLC during a site visit. Constraints: every element of Chapter 23 ("Rules of engagement"). Steps: write the rules of engagement. Expected evidence: one page. Rubric: 3 = scope by identifier with the live PLC flow allowed only with Water's named approval, a read-only or connection-only test (no writes), engineer on site, marked test address, communication to control room and SOC, explicit stop conditions, result storage; 2 = one element missing; 1 = informal; 0 = none.

EX068 — How many more tests? Objective: plan sample size. Inputs: Rule C at 8/8. Constraints: use the Wilson lower bound from Lab 23. Steps: find the number of additional consecutive successes needed for the lower bound to reach 0.70. Expected evidence: the number and the run. Rubric: 3 = one more success (9/9 gives a lower bound of about 0.701; 10/10 gives about 0.722), correctly shown, with the comment that meeting a threshold by a margin of 0.001 is fragile and a second extra test is worth planning; 2 = right with unclear method; 1 = guessed; 0 = none.

EX069 — Diagnose a miss. Objective: turn results into work. Inputs: Rule B missed 3 of 10 tests because one station switch's flows were not exported. Constraints: fix and re-test plan. Steps: (1) state the fix; (2) state the monitoring that would have detected the gap earlier (Chapter 19); (3) plan the re-test. Expected evidence: half page. Rubric: 3 = enable flow export on that switch under change; add a source-absence rule per flow exporter; re-test with at least 10 events across all switches; update control card; 2 = fix without monitoring; 1 = "tune rule B"; 0 = none.

Chapter synthesis

Test controls against the evidence they promise, using safe, marked test events under signed rules of engagement that protect the physical process. Fix the claim rule before testing, report counts with intervals and time measures, and distinguish "shown effective" from "not yet shown" — and why. Turn every result into improvement or more tests, and re-test on a schedule.

Continue studying — chapter 23 of 30

← Chapter 22 · Course chapter index · Chapter 24 →

Continue to the next authored lesson.

Worked answers

EX067 — Rules of engagement. Purpose: test Rule B detection. Scope: test workstation TW-01 (marked address 10.40.99.99), spare Arlen 300 in the FSTS lab, and — only during the visit on a named date — one TCP connection test from the engineering laptop to PS-04's PLC on tcp/502, opening and closing a connection with no Modbus writes or reads, with Water's named approval. Out of scope: every other station and controller. Time window: stated hours. On site: Water engineer at PS-04. Communication: Water control room and SOC told before, at start and end; test events labelled "TEST-RULEB" in the change calendar. Stop conditions: any unexpected PLC behaviour, alarm or request from the control room. Results: stored in the evidence store with raw alert output and times.

EX068 — How many more tests? One more consecutive success: 9/9 gives a Wilson lower bound of about 0.7008, just above 0.70; 10/10 gives about 0.7225. Passing a threshold by 0.001 is fragile — a single failure in the next tests would drop it again — so plan at least two more tests and report the interval, not just "shown effective".

EX069 — Diagnose a miss. Fix: enable flow export on the station switch that was not exporting (normal change with Water), verify its flows appear in the SIEM. Monitoring that would have found it earlier: a per-exporter absence rule (Chapter 19) and an inventory check that every switch in the station zone is an active flow exporter. Re-test: at least 10 marked test flows spread across all station switches, under updated rules of engagement; update Rule B's control card with the new result and the cause of the earlier misses.