Chapter 35 — Design studio II
Lesson map
Lesson 35 published. Map, example, EX103.
flowchart TD
N0["Learning outcomes"]
N1["35.1 Brief synthetic"]
N2["35.2 What is new compared with Studio I"]
N3["35.3 Multidisciplinary panel"]
N4["35.4 Iteration"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX103"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["35.1 Brief synthetic"]
N2["35.2 What is new compared with Studio I"]
N3["35.3 Multidisciplinary panel"]
N4["35.4 Iteration"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX103"]
[Learning outcomes]
|
v
[35.1 Brief synthetic]
|
v
[35.2 What is new compared with Studio I]
|
v
[35.3 Multidisciplinary panel]
|
v
[35.4 Iteration]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX103]
Learning outcomes
You will (1) design a complete AI-enabled system with security, safety and operational requirements; (2) produce a safety-and-security case; (3) defend the design before a multidisciplinary panel; and (4) iterate from critique to a revised baseline.
35.1 Brief (synthetic)
Design "Yardmind", an AI system for NRS that uses TrackEye detections, Pulse alerts and Wrench to plan and dispatch maintenance crews across a region, including proposing track speed restrictions to a human controller.
Yardmind combines the three earlier systems, so their weaknesses can compound: a poisoned TrackEye label, a spoofed Pulse reading or an injected bulletin reaching Wrench could each influence crew dispatch. Speed restrictions raise the stakes, since a wrongly lifted restriction can endanger trains directly. The brief asks teams to show how far each upstream weakness can travel and where it is stopped.
35.2 What is new compared with Studio I
Studio II adds a safety-and-security case: a structured argument that the system is acceptably safe and secure, supported by evidence. A common structure is claim → argument → evidence: "Yardmind cannot impose or lift a speed restriction without a controller's approval" (claim), "all restriction actions pass the enforcement point which requires a signed controller decision" (argument), "code review CR-112, test report TR-208, penetration test PT-05" (evidence). Include assumptions and defeaters — what would make the argument fail.
Defeaters keep the case honest by naming what would break each claim.
35.3 Multidisciplinary panel
The panel includes a controller (operational realism), a safety assessor (hazard coverage), a red-teamer (attack paths), a data-protection officer (personal data of crews) and a finance lead. Each signs off or lists conditions.
Each perspective sees a different risk. The controller knows that a proposal arriving during a busy shift will be approved quickly; the safety assessor checks whether every hazard has a control; the red-teamer looks for the weakest input; the data-protection officer asks why crew locations are stored and for how long; the finance lead asks whether controls fit the budget. Conditions from one role sometimes conflict with another, and resolving them is part of the exercise.
35.4 Iteration
Teams revise within one week and present a delta: what changed, why, and which panel conditions are now met. Unresolved conditions become explicit residual risks with owners.
Presenting a delta keeps the second review focused. The panel does not re-read the whole design; it checks each condition against the change made and the evidence offered. Conditions that cannot be met within the week are not hidden: they are recorded with an owner and a date, and the safety-and-security case shows them as open defeaters.
35.5 Worked example (synthetic NRS case)
Sam's team argued that Yardmind could propose speed restrictions but never apply or lift them. The red-teamer asked about lifting: could a manipulated Pulse feed cause Yardmind to recommend lifting a restriction prematurely? The team added a rule: recommendations to lift restrictions require two independent evidence sources (a TrackEye pass and a human inspection record), and controllers see both. The safety assessor accepted with the condition that lifting recommendations are logged for monthly audit.
The rule removes a single point of failure: a spoofed Pulse feed alone cannot trigger a lifting recommendation, because a TrackEye pass and a human record are also needed.
35.6 Errors and limits
- Safety cases without defeaters.
- Treating "propose" as risk-free when humans routinely accept proposals.
- Ignoring crew personal data in dispatch.
A case without defeaters looks complete but cannot be challenged, so its weaknesses stay hidden until an incident. 'Propose' is only safe if humans review proposals with care, and the approval measurements in the oversight chapter showed how quickly that can fail. Crew dispatch involves locations, shift patterns and names, so data protection is a design input, not a later review.
Exercises
- EX103 — Claim–argument–evidence. Write one claim–argument–evidence triple for Yardmind. Rubric: 3 = specific claim, sound argument, concrete evidence items; 2 = one weak; 1 = vague; 0 = none.
- EX104 — Defeater. Name a defeater for the claim in 35.2 and how to address it. Rubric: 3 = e.g., forged controller signature or compromised enforcement point; key protection/independent monitor; 2 = defeater only; 1 = vague; 0 = none.
- EX105 — Lifting rule. Why are lifting recommendations treated more strictly than imposing ones? Rubric: 3 = lifting removes protection; error/attack consequences are higher; asymmetric risk; 2 = partial; 1 = vague; 0 = wrong.
Synthesis
Studio II makes the design argue for itself: a safety-and-security case with evidence and defeaters, tested by a multidisciplinary panel and revised to explicit residual risk.
Continue studying — chapter 35 of 50
← Chapter 34 (draft) · Course chapter index · Chapter 36 (draft) →