Chapter 38 — Critical infrastructure and sector security: validation | Engineering Critical infrastructure and sector security

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 38 published. Map, example, EX075.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["38.1 Verification is not validation"]
  N2["38.2 Traceability as an evidence map"]
  N3["38.3 Stakeholder validation"]
  N4["38.4 Reporting partial results"]
  N5["38.6 Practical activity: Lab 14"]
  N6["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX075"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["38.1 Verification is not validation"]
  N2["38.2 Traceability as an evidence map"]
  N3["38.3 Stakeholder validation"]
  N4["38.4 Reporting partial results"]
  N5["38.6 Practical activity: Lab 14"]
  N6["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX075"]

[Learning outcomes and connection]
    |
    v
[38.1 Verification is not validation]
    |
    v
[38.2 Traceability as an evidence map]
    |
    v
[38.3 Stakeholder validation]
    |
    v
[38.4 Reporting partial results]
    |
    v
[38.6 Practical activity: Lab 14]
    |
    v
[Chapter synthesis]
    |
    v
[Worked example] --> [Exercise EX075]

Learning outcomes and connection

After this chapter you can (1) distinguish validation (right system for the need) from verification (system built to its specification); (2) validate with stakeholders through demonstrations, operational trials and exercises; (3) use traceability to show which needs are supported by evidence and which are not; and (4) report validation results honestly, including partial results. It builds on Chapters 16, 24 and 30 and prepares Chapters 40 and 48.

38.1 Verification is not validation

A system can pass every test against its specification and still fail its users if the specification was wrong or incomplete. Validation asks whether the system meets stakeholder needs in its intended environment: can operators actually run local manual mode at night? Does the hospital actually get water within its buffer when PS4 is lost? Validation evidence comes from realistic use: operational trials, demonstrations with stakeholders, exercises and early operating data.

For TMRP, the night exercise of local manual mode is the validation activity that answers the first question.

38.2 Traceability as an evidence map

Trace needs to requirements to tests to results. Then ask three questions: which requirements have no test (untested claims)? Which tests failed or were not run (unsupported claims)? Which needs depend only on requirements that are not yet verified (unvalidated needs)? Lab 14 computes these for the TMRP dossier.

For TMRP the questions are concrete. R4 has no test because its demonstration was never scheduled. R9, the independent dosing alarm, has no test at all: the requirement table names a test as its verification method, but no test was ever written for it. R3's only recorded test is the failed one on v1, and R8's test has not run. N1 depends on R4, so it cannot be claimed as validated until the night exercise takes place. Lab 14 prints these gaps directly.

38.3 Stakeholder validation

Involve the people the system serves. Operators validate procedures and displays; maintenance staff validate repairability and spares; the hospital validates that the supply arrangements meet its needs; regulators validate compliance reasoning. Record their findings as you would review findings, with responses.

For TMRP, operators validate the local manual procedure and the new security alarms, maintenance staff check that gateways can be replaced from spares, and the hospital confirms that the supply arrangements and its four-hour buffer meet its needs. Their findings go into the same response log as the design review findings.

38.4 Reporting partial results

Report validation status per need: validated, partially validated (with what remains), not validated. Do not round up. "N1 partially validated: PS4 loss trial met the 4 h buffer; combined PS4 and grid F1 loss not yet trialled."

TMRP's evidence map makes this concrete: N1 depends on R4, which has not yet been demonstrated, so N1 is reported as not validated until the night exercise takes place, even though most of its other requirements have passed their tests. The status is updated after the exercise.

38.5 Worked example: TMRP evidence map (synthetic)

Lab 14 shows: R4 (local manual operation) has no test — it was planned as a demonstration and never scheduled; R3 has a failed test (on v1; the re-test on v2 has not yet been entered into the matrix); R8's test T6 has not run. R9 (independent dosing alarm) has no test either: its verification method is "Test", but no test identifier was ever assigned, so the evidence for it is a design intent, not a result. Verified: 5 of 9 requirements (55.6%). Consequence for validation: N1 (hospital supply) depends on R4, so N1 cannot be claimed as validated; the night-time local manual exercise is scheduled as the validation activity. The matrix also exposes an administrative gap — the v2 re-test of T3 must be recorded — which is exactly the kind of inconsistency examiners and auditors find.

38.6 Practical activity: Lab 14

# Lab 14 - requirements traceability: what is claimed but not shown? (synthetic TMRP dossier, offline)
REQS = ["R1 auth telemetry", "R2 replay reject", "R3 setpoint bounds", "R4 local manual", "R5 24h hold-up",
        "R6 restore 8h", "R7 alarm 60s", "R8 log integrity", "R9 dosing alarm"]
TESTS = {"T1": ["R1"], "T2": ["R1", "R2"], "T3": ["R3"], "T4": ["R5"], "T5": ["R7"], "T6": ["R8"], "T7": ["R6"]}
RESULT = {"T1": "pass", "T2": "pass", "T3": "fail", "T4": "pass", "T5": "pass", "T6": "not run", "T7": "pass"}
ids = [r.split()[0] for r in REQS]
cov = {r: [t for t, rs in TESTS.items() if r in rs] for r in ids}
verified = [r for r in ids if cov[r] and all(RESULT[t] == "pass" for t in cov[r])]
print("untested:", [r for r in ids if not cov[r]])
print("tested but not passing:", [r for r in ids if cov[r] and r not in verified])
print(f"verified {len(verified)}/{len(ids)} = {100*len(verified)/len(ids):.1f}%")

Expected output:

untested: ['R4', 'R9']
tested but not passing: ['R3', 'R8']
verified 5/9 = 55.6%

The output shows that R4 and R9 have no test, that R3 and R8 are tested but not passing, and that five of nine requirements are verified. R3's status is out of date rather than truly failed: the v2 re-test passed, but nobody entered it in the matrix, which is exactly the administrative gap the worked example warns about. EX075 updates the map. R9 is the harder lesson: the dosing alarm appears in the requirement set, the risk table and the dosing controls, so it feels covered, yet no test was ever assigned to it. An earlier version of this matrix left R9 out of the list entirely, which raised the verified share from five of nine to five of eight without any new evidence. A matrix must list every defined requirement, because a requirement it omits cannot be reported as untested.

38.7 Common errors and limits

Calling a passed test suite "validation".
Stakeholders consulted only at the start.
Rounding partial validation up to "validated".
Evidence maps not updated after re-tests.

TMRP's map shows the last error itself: the T3 re-test on v2 passed, but the matrix was never updated. Stakeholders such as operators and the hospital are involved throughout, not only when the requirements were written, and the evidence map is updated every time a re-test or demonstration produces a result, so the reported validation status never runs ahead of the evidence.

Exercises — chapter 38

EX075 — Update the map. Objective: maintain traceability. Inputs: Lab 14. Steps: (1) add T3b (v2 re-test) covering R3 with result "pass" and remove T3's link to R3 as superseded; (2) add T8 demonstration covering R4 with result "pass"; (3) run T6 with result "pass"; recompute. Expected evidence: code and output. Rubric: 3 = eight of nine verified (88.9%), R9 still listed as untested because no dosing-alarm test exists, with a note that verification ≠ validation; 2 = one change missed; 1 = not run; 0 = none.

EX076 — Validation plan for N1. Objective: plan stakeholder validation. Inputs: N1 "hospital must not lose water > 4 h". Steps: propose two validation activities with stakeholders, realistic conditions, observations and success criteria. Expected evidence: plan. Rubric: 3 = realistic activities (night exercise of local manual with real staffing; planned isolation of PS4 with hospital monitoring), criteria tied to the 4 h buffer, stakeholders named by role; 2 = criteria vague; 1 = tests renamed as validation; 0 = none.

Chapter synthesis

Verification asks whether the system was built right; validation asks whether it is the right system. Use traceability as an evidence map to find untested, unsupported and unvalidated claims, validate with the people the system serves under realistic conditions, and report partial results as partial.