Chapter 23 — OT, IoT and cyber-physical security: evaluation
Lesson map
Lesson 23 published. Map, example, EX067.
flowchart TD
N0["Learning outcomes"]
N1["23.1 Did it work?"]
N2["23.2 Tests with expected results"]
N3["23.3 Before and after"]
N4["23.4 Honest reporting"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX067"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["23.1 Did it work?"]
N2["23.2 Tests with expected results"]
N3["23.3 Before and after"]
N4["23.4 Honest reporting"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX067"]
[Learning outcomes]
|
v
[23.1 Did it work?]
|
v
[23.2 Tests with expected results]
|
v
[23.3 Before and after]
|
v
[23.4 Honest reporting]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX067]
Learning outcomes
You will (1) evaluate whether controls achieved their purpose; (2) design simple tests with expected results; (3) measure before and after; and (4) report effectiveness honestly, including failures.
23.1 Did it work?
After six months of changes at NCS — rule 9 removed, supervised contractor sessions, camera zone, certificate automation — management asks whether the site is "more secure". An evaluation answers narrower, testable questions: is the office → control path closed? Do contractor sessions require approval? Are certificates renewed before expiry?
Each question maps to a control in Priya's register and to a test that can fail. Asking "is NCS more secure?" would invite a yes that nobody could check; asking whether OFFICE-7 can still reach PLC-R2:502 invites a refused connection in a firewall log, which anyone can check.
23.2 Tests with expected results
For each control, write the test and the expected result before running it: "From OFFICE-7, attempt a TCP connection to PLC-R2:502 in the maintenance window; expected: refused at the firewall and logged." "Contractor attempts to connect without approval; expected: jump host denies." Tests that touch the control zone are run in maintenance windows with operations informed.
Priya's table had six such tests with expected results written first. Five passed. The sixth found the camera-zone rule letting CAM-DOCK reach HIST on a web port, because a broad "camera to server" rule had been written instead of one naming the video recorder.
23.3 Before and after
Measures that existed before changes give a comparison: unapproved flows per week (before: 2 types; after: 0), HMI accounts belonging to leavers (before 2; after 0), days of logger data lost to expired certificates (before 7 in one incident; after 0 in six months).
All three measures existed before the changes because Priya's earlier work produced them: the Chapter 12 flow comparison counted the unapproved flows, the Chapter 7 review counted the leaver accounts and the Chapter 22 check dated the LOG-03 gap. Without that baseline, the six-month evaluation could only have said that the problems were not seen now.
23.4 Honest reporting
Report what failed. If one contractor still used the cellular router (Chapter 14) because it was switched on for an emergency and never switched off, say so, with the corrective action. Evaluations that report only successes stop being believed.
Priya's evaluation reports its one failure the same way: the camera-zone rule allowed CAM-DOCK to reach HIST on a web port; the corrective action narrows the rule to the video recorder only, and a retest is scheduled. Reporting five of six as five of six is what makes the five believable.
Worked example (synthetic NCS)
Priya's evaluation table: six controls, six tests, five passed. The failed row is reported in full, with its cause, corrective action and retest date.
Synthetic case — Northgate Cold Storage. Results are invented.
Errors and limits
A passed test shows the control worked at test time for that case; drift can undo it. Before-and-after counts can change for reasons other than the control.
Exercises
- EX067 — Writing a test. Write the test and expected result for "only HMI-C and HIST can reach PLC-R1". Rubric: 3 = test from a non-approved host plus approved host, expected outcomes, window; 2 = one side; 1 = vague; 0 = none.
- EX068 — Interpreting a before/after count. Unapproved flows fell from 2 types to 0. Name one reason other than the controls. Rubric: 3 = observation window/capture placement changed or seasonal activity absent; 2 = vague; 1 = minimal attempt; 0 = none.
- EX069 — Reporting the failure. Write the evaluation line for the camera rule failure. Rubric: 3 = test, result, cause, corrective action, retest date; 2 = three; 1 = "minor issue"; 0 = none.
Synthesis
Evaluation turns "more secure" into specific tests with expected results, compared before and after, with failures reported as plainly as passes.
Continue studying — chapter 23 of 30
← Chapter 22 (draft) · Course chapter index · Chapter 24 (draft) →