Chapter 25 — Design studio I
Lesson map
Lesson 25 published. Map, example, EX073.
flowchart TD
N0["Learning outcomes"]
N1["25.1 Studio format"]
N2["25.2 The brief synthetic"]
N3["25.3 The design package"]
N4["25.4 Critique"]
N5["25.4b Trade-off tables"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX073"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["25.1 Studio format"]
N2["25.2 The brief synthetic"]
N3["25.3 The design package"]
N4["25.4 Critique"]
N5["25.4b Trade-off tables"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX073"]
[Learning outcomes]
|
v
[25.1 Studio format]
|
v
[25.2 The brief synthetic]
|
v
[25.3 The design package]
|
v
[25.4 Critique]
|
v
[25.4b Trade-off tables]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX073]
Learning outcomes
You will (1) work in a team to design an AI-enabled subsystem with security requirements; (2) present design alternatives and trade-offs; (3) receive and respond to design critique; and (4) produce a design package (requirements, architecture, threat model, verification plan).
25.1 Studio format
Teams of three to five receive a brief, produce a design in two weeks, present it to a panel and revise it. The panel plays roles: operator, attacker, safety assessor and budget owner. The point is to practise making and defending trade-offs, not to find a perfect design.
The roles pull in different directions on purpose. An operator wants speed and few clicks, an attacker wants any path to impact, a safety assessor wants failures to be safe, and a budget owner wants cost justified. A design that satisfies one role completely usually fails another, and the panel's questions expose that before any code is written.
25.2 The brief (synthetic)
Design an LLM-based assistant that helps NRS depot planners schedule inspections using maintenance history, weather forecasts and supplier bulletins. It may propose schedule changes; a planner must approve them.
The brief is small but contains the course's central tension. Maintenance history is internal, weather forecasts come from an outside feed, and supplier bulletins are written by third parties, so three trust levels meet in one assistant. The assistant may only propose, yet proposals to postpone inspections affect rail safety. Teams must decide how far each source may influence a proposal and what a planner needs to see before approving it.
25.3 The design package
- Context and stakeholders. 2. Requirements (functional, security, statistical, operational) with verification methods. 3. Architecture with trust boundaries and an external policy point. 4. Threat model ranked and mapped to OWASP/ATLAS. 5. Verification plan: evaluation suites, sample sizes, gates. 6. Residual risks and open questions.
The order mirrors how a reviewer reads: who the system is for, what it must do, how it is built, what could go wrong, how that will be checked and what remains. Each item references the earlier ones, so a threat points to the requirement it endangers and the test that will verify the control. A package missing its verification plan is returned unread, because without it nothing else can be checked.
25.4 Critique
Each panel role asks questions from its perspective. The attacker role is especially valuable: "I control a weather-feed mirror. What can I do?" Teams must answer with their architecture, not with promises about the model.
An answer such as 'the model will ignore malicious instructions' fails, because it relies on behaviour nobody can guarantee. An acceptable answer points to the architecture: the weather feed's values pass through a validator that accepts only numbers within physical limits, and any schedule change still needs a planner's approval with the source shown. Teams learn to welcome hard questions, since each one found in the studio is one less found in service.
25.4b Trade-off tables
Present alternatives in a table with columns for security effect, safety effect, user friction, cost and residual risk. A panel can then see why an option was rejected rather than guessing. For example, "no tool access" has the lowest risk but loses most of the benefit; "free tool access" has the highest benefit and unacceptable risk; "proposal plus approval" sits between. Being explicit about what each option gives up is more persuasive than claiming any option is free.
Sam's planner-assistant revision fits in such a table: "bulletins read directly into the prompt" was rejected for its residual risk, while "summarised by a low-privilege model plus two approvals for safety-critical changes" accepted some extra planner friction for a bounded worst case.
25.5 Worked example (synthetic NRS case)
Sam's team proposed reading supplier bulletins directly into the planner prompt. The attacker panelist showed (in design terms) that a bulletin could ask the model to postpone inspections on a route. The team revised: bulletins are summarised by a separate low-privilege model whose output is labelled untrusted; schedule changes affecting safety-critical inspections require two approvals; and every proposed change shows the source documents that influenced it.
The revision shows the pattern the studio aims to teach: the attack was answered by changing structure, not by improving the prompt.
25.6 Errors and limits
- Designs that rely on the model "ignoring" malicious text.
- Packages without verification plans.
- Defensive responses to critique.
Relying on the model to ignore malicious text is a promise, not a control. A package without a verification plan cannot show that any requirement is met. Defensive responses, such as arguing that an attacker would never try something, waste the panel's most useful feedback. Sam's team adopted a rule for the studio: every critique gets a written response saying accepted, rejected with reason, or deferred with an owner and date.
Exercises
- EX073 — Package. List the six parts of the design package. Rubric: 3 = all six; 2 = four; 1 = two; 0 = none.
- EX074 — Attacker question. Answer the weather-feed mirror question for your design. Rubric: 3 = identifies influence path and architectural limit; 2 = partial; 1 = relies on model; 0 = none.
- EX075 — Two approvals. Why require two approvals only for safety-critical changes? Rubric: 3 = proportional control, limits friction while bounding high-impact risk; 2 = partial; 1 = vague; 0 = wrong.
Synthesis
The studio practises the core engineering skill: making explicit trade-offs and defending them against operators, attackers, safety assessors and budget owners.
Continue studying — chapter 25 of 50
← Chapter 24 (draft) · Course chapter index · Chapter 26 (draft) →