Chapter 25 — Team project studio | Technologist AI and machine-learning security

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 25 — Team project studio

Lesson map

Lesson 25 published. Map, example, EX073.

flowchart TD
  N0["Learning outcomes"]
  N1["25.1 The studio brief"]
  N2["25.2 Roles"]
  N3["25.3 Peer review"]
  N4["25.4 Stakeholder presentation"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX073"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["25.1 The studio brief"]
  N2["25.2 Roles"]
  N3["25.3 Peer review"]
  N4["25.4 Stakeholder presentation"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX073"]

[Learning outcomes]
    |
    v
[25.1 The studio brief]
    |
    v
[25.2 Roles]
    |
    v
[25.3 Peer review]
    |
    v
[25.4 Stakeholder presentation]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX073]

Learning outcomes

You will (1) work in a team to secure an AI feature end to end; (2) divide roles and coordinate work; (3) review each other's security work; and (4) present results to stakeholders.

25.1 The studio brief

Teams of four receive a synthetic AI feature on a staging environment — for example, a returns assistant with two tools — and six weeks. Deliverables: threat model, controls, evaluation harness and results, runbooks and a presentation. Each team keeps a shared decision log so that reasons for choices survive the project. Jonas's team received a delivery-rescheduling assistant as its brief, with the same staging platform as Ask Brightwell. Its decision log recorded, for example, why rescheduling rules were placed in a policy service rather than the prompt.

25.2 Roles

Rotate roles weekly so everyone practises each: architect (diagrams, design decisions), builder (controls and code), tester (suites and harness, independent from the builder), operator (monitoring, runbooks). The tester role must not test their own code that week; independence is the point. In Jonas's team, Mei was the tester in the week she found the cancellation-window gap, and she had not written the policy code. That independence is why she looked for ways around the rules instead of confirming that they worked as intended.

25.3 Peer review

Every control is reviewed by someone who did not write it. Reviews check: does the code enforce what the threat model says? Are there tests? Are defaults safe? Is there a way around it? Reviewers write comments that name the issue, why it matters and a suggested fix. In Jonas's team, the rescheduling rules were reviewed by a teammate who had not written them. The review asked whether the rules matched the threat model's scenario about blocking cancellation, which is how the missing date check became a named issue with a suggested fix.

25.4 Stakeholder presentation

Ten minutes: what the feature does, top risks, controls, evidence (with intervals), residual risk and the decision you recommend. Expect questions from a product owner (value), a support lead (usability) and a security lead (evidence). Jonas's team's presentation of the rescheduling assistant ended with a residual-risk slide: changes made by phone bypass the assistant's rules. The support lead asked how staff would know, which led to a note in the support runbook. The security lead asked how the 0 of 60 result was graded, and the team showed the tool-log grader.

25.5 Worked example (synthetic Brightwell case)

Jonas's team secures a "delivery rescheduling" assistant. The tester that week, Mei, writes adaptive cases without seeing the policy code and finds that a customer can reschedule a delivery to a date after the order's cancellation window, effectively blocking cancellation. The builder adds the rule; the harness confirms 0 of 60 adaptive cases succeed afterwards. The team presents with a residual-risk slide: phone-channel changes bypass the assistant's rules. The finding came from an independent tester, and the fix was confirmed by the harness rather than by the builder's word.

25.6 Errors and limits

One person doing all the security work.
Testers reviewing their own code.
Presentations without residual risk.

Jonas's team avoided these by rotating roles, keeping the tester independent of that week's code, and presenting the phone-channel bypass as residual risk. One person doing all the work would have tested only what they built. A presentation without residual risk would have let owners assume phone changes were covered. The weekly role rotation is what keeps each of these from happening, because nobody holds the same role two weeks in a row.

Exercises

EX073 — Roles. Explain why the tester role must be independent. Rubric: 3 = blind spots, assumptions shared with builder, evidence credibility; 2 = two; 1 = one; 0 = none.
EX074 — Review comment. Write a review comment on a policy rule that lacks a default deny. Rubric: 3 = issue, impact, fix; 2 = two; 1 = vague; 0 = none.
EX075 — Presentation. Outline the ten-minute presentation for Mei's team. Rubric: 3 = feature, risks, controls, evidence, residual, decision; 2 = four or five; 1 = two; 0 = none.

Synthesis

Team studios teach security as a shared practice: rotating roles, independent testing, peer review and honest presentation.

Continue studying — chapter 25 of 30

← Chapter 24 (draft) · Course chapter index · Chapter 26 (draft) →