Chapter 25 — Team project studio
Lesson map
Lesson 25 published. Map, example, EX073.
flowchart TD
N0["Learning outcomes and connection"]
N1["25.1 The team and its roles"]
N2["25.2 Work items with acceptance criteria"]
N3["25.3 Review as a control"]
N4["25.4 Rhythm"]
N5["25.5 Blameless reviews"]
N6["25.7 Practical activity: project backlog"]
N7["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX073"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["25.1 The team and its roles"]
N2["25.2 Work items with acceptance criteria"]
N3["25.3 Review as a control"]
N4["25.4 Rhythm"]
N5["25.5 Blameless reviews"]
N6["25.7 Practical activity: project backlog"]
N7["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX073"]
[Learning outcomes and connection]
|
v
[25.1 The team and its roles]
|
v
[25.2 Work items with acceptance criteria]
|
v
[25.3 Review as a control]
|
v
[25.4 Rhythm]
|
v
[25.5 Blameless reviews]
|
v
[25.7 Practical activity: project backlog]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX073]
Learning outcomes and connection
After this chapter you should be able to:
- work as part of a small delivery team building and operating a shared service;
- break the LANTERN integration project into work items with acceptance criteria and evidence;
- use review — of code, configuration and documents — as a security control;
- run short, blameless reviews of what went well and badly, and turn them into changes.
From Chapter 26 onwards the book moves to hand-off, deployment and the capstone. Real services are built by teams, and many security failures are team failures: an assumption nobody checked because each person thought another had, a change merged without review because the reviewer was busy. This chapter is a studio: it gives you a structure for the team work that the capstone needs.
25.1 The team and its roles
A LANTERN delivery team at FSTS has four or five people: a service owner (prioritizes, speaks for customers), two or three technologists (build and operate), a security reviewer (may be one of the technologists, rotated) and a customer representative from Water operations who attends reviews and accepts work that affects the process. Roles rotate where possible so that knowledge is shared; no single person should be the only one who understands the relay or holds its keys (Chapter 8's independence).
25.2 Work items with acceptance criteria
Every work item states what "done" means as acceptance criteria that produce evidence. "Add station-silence alerting" is a wish. "Rule C alerts within 20 minutes of a station's heartbeat stopping, verified by three injected test silences across two stations; first action written and agreed by Water control room; control card updated" is a work item. The evidence becomes part of the capstone record.
The NTP merge in this chapter shows the cost of a weak criterion: "update NTP servers" was met while every relay lost its second time source. The rewritten criterion, "offset < 1 s on all relays for 24 h", would have failed on the first day.
25.3 Review as a control
Peer review of code, desired-state files, firewall changes and runbooks catches errors, spreads knowledge and makes it harder for one compromised or careless account to change the service alone. Make review required for anything that changes the service (protected branches, Chapter 22), specific (the reviewer checks against acceptance criteria and the design properties of Chapter 8, not just style), and recorded. Rotate reviewers so that review is not rubber-stamped by one habitual approver.
The NTP merge was reviewed, but for syntax; the reviewer never checked that the new server existed.
25.4 Rhythm
A short, regular rhythm suits operational teams: a weekly planning conversation (what will be done, by whom, with what evidence), a brief daily check (what is blocked), and a review at the end of each cycle (what was delivered, shown with its evidence to the customer representative). Operational work — incidents, faults — interrupts project work; plan for it rather than pretending it will not happen.
The PS-09 silence and the jump-host disk failure are the interruptions to plan for.
In Fallowmere, the daily check is where the PS-09 silence was raised the morning after Rule C fired.
25.5 Blameless reviews
When something goes wrong — a failed change, a missed alert, a near-miss — hold a short blameless review: what happened, what people knew at each moment, why their actions made sense to them then, what conditions allowed the problem, and what will change. Google's SRE material describes the value of blameless postmortems for learning from incidents without discouraging honest reporting (Google SRE book, postmortem culture); Reason's work on human error distinguishes blaming individuals from fixing the system conditions that make errors likely (Reason, 2000). The output is changes, each with an owner — not a person to blame.
25.6 Worked example: a merge that should not have happened (synthetic)
During the project, a technologist merged a desired-state change that removed 10.30.0.2 from the relays' NTP servers, intending to replace it with a new server that had not yet been built. The automation applied it; relays kept one time source; when that source restarted for maintenance two days later, relays drifted, and station messages began to fail the 120-second window. The blameless review found: the change was reviewed, but the reviewer checked syntax, not whether the new server existed; the acceptance criteria said only "update NTP servers"; drift monitoring checked that settings matched desired state, which they did; and there was no alert on clock offset. Changes: review checklist includes "does every referenced system exist and respond?"; acceptance criteria for time changes include "offset < 1 s on all relays for 24 h"; new alert on relay clock offset > 2 s. Nobody was blamed; three conditions were fixed.
25.7 Practical activity: project backlog
Write the backlog for the capstone (Chapter 29): at least ten work items covering the relay, validation, authentication, historian, logging, monitoring rules, access, backup and restore, automation, runbooks and evidence. Each needs acceptance criteria that produce evidence and an owner role.
One complete item as a model: "Relay clock offset alert: alert when any relay's offset exceeds 2 s; acceptance, an injected offset on a test relay raises the alert within one evaluation cycle and the first action is written; owner, FSTS monitoring." It comes straight from the NTP review. Include at least one item that exists only to close a gap found earlier, such as FIM's 1,440-minute median detection time.
25.8 Common errors and limits
- Work items without acceptance criteria.
- Review that checks style, not substance; or one habitual reviewer.
- A single team member as the only person who understands a component.
- Plans that ignore operational interruptions.
- Reviews of failures that end with "be more careful".
The NTP review in this chapter ended with three changed conditions, not with "be more careful".
The NTP merge shows the first two: a vague acceptance criterion and a review of syntax only. A single person understanding the key store is the third, which the handover in Chapter 26 exposed when the receiving team could not find it.
Exercises — chapter 25
EX073 — Acceptance criteria. Objective: write evidence-producing criteria. Inputs: work items: (a) "secure SSH on relays"; (b) "backups for historian"; (c) "vendor access process". Constraints: each criterion must be checkable by someone else. Steps: rewrite each with at least three criteria. Expected evidence: list. Rubric: 3 = for example (a) password auth off and listener on management interface on all relays (drift check output), connection from corporate zone refused (test record), MFA on jump host; (b) nightly backup with one immutable copy, restore test within RTO recorded; (c) time-bound enablement tested end to end, expiry verified, Kell acknowledged; 2 = criteria partly checkable; 1 = restated wishes; 0 = none.
EX074 — Review checklist. Objective: make review substantive. Inputs: the Chapter 25 ("Worked example: a merge that should not have happened") case and Chapter 8 design properties. Constraints: eight items. Steps: write a checklist for reviewing a desired-state change. Expected evidence: checklist. Rubric: 3 = includes existence and reachability of referenced systems, direction of trust preserved, no new listener on all interfaces, secrets not in the change, rollback defined, acceptance criteria and verification, affected customers informed, staged rollout; 2 = six; 1 = style-only items; 0 = none.
EX075 — Blameless review. Objective: run a review that produces changes. Inputs: incident: an on-call technologist silenced Rule B for 24 hours during a noisy night and forgot to unsilence it; a real undeclared flow went unalerted for three days. Constraints: blameless; conditions, not people. Steps: write the review: timeline, what made sense at the time, conditions, changes with owners. Expected evidence: one page. Rubric: 3 = recognizes Rule B's per-event noise (Chapter 19) as a cause, no automatic expiry on silences, no visibility of silenced rules; changes such as episode alerting, silences that expire and are shown on the dashboard, handover checklist item; 2 = changes partial; 1 = blames the technologist; 0 = none.
Chapter synthesis
Services are built by teams. Give roles clear responsibility and rotate them so that knowledge and control are not concentrated in one person. Write work items with acceptance criteria that produce evidence. Make review required, specific and recorded, so that it acts as a security control. Plan for operational interruptions, and learn from failures with blameless reviews that change conditions rather than blame people.
Continue studying — chapter 25 of 30
← Chapter 24 · Course chapter index · Chapter 26 →
Continue to the next authored lesson.