◇ OpenCyber Academy
Engineering · Chapter 45 · English

Chapter 45 — Independent design review

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 45 — Independent design review

Lesson map

Lesson 45 published. Map, example, EX133.

flowchart TD
  N0["Learning outcomes"]
  N1["45.1 Independence"]
  N2["45.2 Review method"]
  N3["45.3 Findings"]
  N4["45.4 Receiving a review"]
  N5["45.4b Severity criteria"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX133"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["45.1 Independence"]
  N2["45.2 Review method"]
  N3["45.3 Findings"]
  N4["45.4 Receiving a review"]
  N5["45.4b Severity criteria"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX133"]

[Learning outcomes]
    |
    v
[45.1 Independence]
    |
    v
[45.2 Review method]
    |
    v
[45.3 Findings]
    |
    v
[45.4 Receiving a review]
    |
    v
[45.4b Severity criteria]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX133]

Learning outcomes

You will (1) conduct an independent security design review of another team's AI system; (2) use structured checklists and threat-driven questions; (3) write findings with severity, evidence and recommendations; and (4) respond professionally to a review of your own design.

45.1 Independence

Reviewers must not have designed the system. They need the design package, access to designers for questions, and enough time. Independence reveals assumptions that designers cannot see.

A designer reads the design with knowledge of what was intended and fills gaps without noticing. A reviewer reads only what is written. That difference finds assumptions such as 'retrieval is filtered somewhere' that the designers never wrote down. Access to the designers matters too, but questions and answers are recorded, so the explanation ends up in the design rather than in the reviewer's notes.

45.2 Review method

Combine a checklist and threat-driven exploration. Checklist items: trust boundaries drawn; enforcement outside the model; least-privilege tools; delegated identity; provenance and taint; output handling; supply-chain verification; data protection; monitoring and runbooks; evaluation gates with held-out suites; human-oversight design; failure modes and safe states. Threat-driven: walk OWASP LLM 2025 entries and relevant MITRE ATLAS techniques against the design and ask, for each, "what stops this, and what if that fails?"

The checklist ensures coverage, and the threat walk-through finds problems specific to the design.

45.3 Findings

Each finding: title; severity (with criteria); description; evidence (diagram, document section); impact; recommendation; and status. Separate findings from observations and positive notes. Be specific enough that the designers can act.

A finding such as 'retrieval may be insecure' gives designers nothing to act on. 'Retrieval uses a service account with access to all depots; filtering happens only in the prompt; a user could ask for another depot's documents' states the evidence and impact, and the fix follows naturally. Positive notes matter as well, because they show which practices to keep when the design changes.

45.4 Receiving a review

Thank reviewers, clarify misunderstandings in the design documents (if a reviewer misunderstood, others will), accept or contest findings with evidence, and track every finding to closure.

Contesting a finding is legitimate when evidence supports it, for example a test showing the reviewer's attack path is blocked. Tracking every finding to closure, including contested ones, keeps the review from being forgotten after the meeting. A short response table with each finding, the decision and the evidence is usually enough, and it goes into the dossier.

45.4b Severity criteria

Agree severity criteria before the review. For example: critical — an external attacker can cause safety impact or large-scale disclosure without user interaction; high — impact requires one plausible condition; medium — impact limited or requires unusual conditions; low — hardening. Shared criteria prevent arguments about labels and let findings from different reviews be compared.

Applied to Sam's Crew Chat review, the retrieval finding is high rather than critical: a technician from one depot needs only one plausible condition, a well-phrased question, to reveal another depot's documents, but the attacker must already be an internal user. The missing output escaping is medium, because exploiting it requires attacker-controlled text in a rendered answer.

45.5 Worked example (synthetic NRS case)

Sam reviewed another team's "Crew Chat" design (an assistant for track crews). High finding: the assistant's retrieval used a service account with access to all depots' documents, filtering results only in the prompt ("only answer about your depot"). Recommendation: per-user retrieval permissions enforced in the vector store query. Medium: no output escaping for a web view. Positive: excellent provenance labelling. The team fixed the high finding before the next milestone.

Filtering in the prompt is a request to the model, not access control; a well-phrased question could retrieve and reveal another depot's documents.

45.6 Errors and limits

Self-review rarely finds the assumptions that caused a problem, because the same person made them. Findings without evidence are hard to act on and easy to dismiss. Ignoring an inconvenient review wastes the effort that went into it and leaves the risk in place, often to be rediscovered later in an incident. Sam's team kept every finding in a tracker that the panel could read. Closed items stayed visible.

Exercises

Synthesis

Independent review, using structured checklists plus threat-driven questions, produces specific, evidenced findings that are tracked to closure.

Continue studying — chapter 45 of 50

← Chapter 44 (draft) · Course chapter index · Chapter 46 (draft) →