Chapter 16 — Systems engineering
Lesson map
Lesson 16 published. Map, example, EX046.
flowchart TD
N0["Learning outcomes"]
N1["16.1 The V-model"]
N2["16.2 Requirements decomposition"]
N3["16.3 Interfaces and emergence"]
N4["16.4 Hazard analysis"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX046"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["16.1 The V-model"]
N2["16.2 Requirements decomposition"]
N3["16.3 Interfaces and emergence"]
N4["16.4 Hazard analysis"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX046"]
[Learning outcomes]
|
v
[16.1 The V-model]
|
v
[16.2 Requirements decomposition]
|
v
[16.3 Interfaces and emergence]
|
v
[16.4 Hazard analysis]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX046]
Learning outcomes
You will (1) apply the systems-engineering V-model to AI-enabled systems; (2) decompose system requirements into component requirements with verification methods; (3) manage interfaces and emergent behaviour; and (4) use hazard analysis that includes AI failure modes.
16.1 The V-model
Systems engineering descends from stakeholder needs to system requirements, architecture, component design and implementation, then ascends through component verification, integration testing, system verification and validation against needs. Every requirement on the left has a verification activity on the right. AI components fit into this but need statistical verification methods (rates with intervals) rather than single pass/fail tests.
A pass for TrackEye is a stated rate with an interval, measured on a named test set, rather than a single correct output.
16.2 Requirements decomposition
A system requirement such as "rail defects larger than 10 mm are detected with probability ≥ 0.98 in all operating conditions" decomposes into camera, lighting, model, alerting and human-review requirements. The model's share might be "false-negative rate ≤ 1.5% in each condition stratum with 95% upper bound", leaving margin for other components.
The split is a budget. If the model may miss 1.5%, the human review step and the alerting chain must keep their own losses small enough that the whole chain still meets 0.98. Each component's requirement names its share and how it will be verified, so a shortfall in one place is visible and can be balanced elsewhere rather than discovered at the end.
16.3 Interfaces and emergence
Interfaces carry assumptions: the model assumes images at a given resolution and exposure; the alert system assumes the model's confidence is calibrated. Emergent behaviour arises when components interact — for example, an LLM agent combining two harmless tools to produce a harmful effect. Interface control documents should state data formats, trust levels and failure behaviour.
When the camera supplier changes a sensor, the interface document tells the team which model assumptions to retest. For agents, emergent harm is found by reviewing combinations of tools, not tools alone: reading a document and sending an email are each harmless, but together they can leak the document.
16.4 Hazard analysis
Techniques such as failure modes and effects analysis (FMEA) and system-theoretic process analysis (STPA) look for ways the system can reach hazardous states. Add AI failure modes: distribution shift, adversarial inputs, poisoning, overconfidence, unsafe tool use. STPA's focus on unsafe control actions is a natural fit for agents that issue commands.
FMEA works upwards from component failures and suits TrackEye's perception chain. STPA works downwards from hazards, asking which control actions would be unsafe if given, withheld, given at the wrong time or for too long. For Wrench, an unsafe action is a work order sent to the wrong asset, and STPA asks what would cause the agent to issue it.
16.5 Worked example (synthetic NRS case)
Sam ran an FMEA row for TrackEye: failure mode "missed defect under glare"; effect "defect undetected until next run"; cause "distribution shift"; detection "dawn stratum monitoring"; severity 9, occurrence 4, detection 5, risk priority number 180. A second row for "adversarial marking" had severity 9, occurrence 2, detection 7 (hard to detect), RPN 126. Both exceeded the action threshold of 100.
The marking row scores lower overall because occurrence is rare, yet its detection score is worse, since nobody monitors for deliberate patterns. Exceeding 100 triggers action for both rows, but the actions differ: better dawn data for one, a detection method for the other. Sam records the scores' basis, since occurrence for attacks is an estimate.
16.6 Errors and limits
- Requirements on AI components without statistical verification methods.
- Interfaces that do not state trust levels.
- Hazard analyses that ignore deliberate attack.
A requirement such as 'the model detects defects' cannot be verified, because no finite test proves it; a rate with a bound can. Interfaces without trust levels let untrusted text flow into privileged components unnoticed. Hazard analyses that consider only random failure miss the cases where someone searches for the worst input on purpose, which is often far worse than the typical case the analysis assumed.
Exercises
- EX046 — Decompose. Decompose "Wrench must not create unauthorised work orders" into three component requirements. Rubric: 3 = policy enforcement, confirmation, tool scope/logging, each verifiable; 2 = two; 1 = vague; 0 = none.
- EX047 — RPN. Compute RPN for severity 8, occurrence 3, detection 6 and state whether it exceeds 100. Rubric: 3 = 144, yes; 2 = arithmetic slip; 1 = method; 0 = none.
- EX048 — Emergence. Give an example of emergent harm from two individually safe tools. Rubric: 3 = plausible chain (e.g., search retrieves data + email tool sends it); 2 = vague chain; 1 = single tool; 0 = none.
Synthesis
Systems engineering gives AI security a backbone: decomposed, verifiable requirements, explicit interfaces and hazard analysis that includes both accidental and adversarial failures.
Continue studying — chapter 16 of 50
← Chapter 15 (draft) · Course chapter index · Chapter 17 (draft) →