Chapter 37 — Evidence review seminar
Lesson map
Lesson 37 published. Map, example, EX109.
flowchart TD
N0["Learning outcomes"]
N1["37.1 Format"]
N2["37.2 Checklist"]
N3["37.3 Feedback"]
N4["37.4 Receiving"]
N5["37.5 Compare"]
N6["37.6 Record"]
N7["37.7 Decline with reason"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX109"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["37.1 Format"]
N2["37.2 Checklist"]
N3["37.3 Feedback"]
N4["37.4 Receiving"]
N5["37.5 Compare"]
N6["37.6 Record"]
N7["37.7 Decline with reason"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX109"]
[Learning outcomes]
|
v
[37.1 Format]
|
v
[37.2 Checklist]
|
v
[37.3 Feedback]
|
v
[37.4 Receiving]
|
v
[37.5 Compare]
|
v
[37.6 Record]
|
v
[37.7 Decline with reason]
|
v
[Worked example] --> [Exercise EX109]
Learning outcomes
You will (1) critique a peer dossier; (2) give actionable feedback; (3) receive critique without defensiveness.
37.1 Format
20 minutes present, 20 critique, 10 actions.
The seminar clock is strict: twenty minutes to present evidence, twenty to critique and ten to capture dated actions. Applause that eats the action slot is a facilitation failure. Sam's facilitator card shows the three colours of time remaining. Presenters who run long lose critique depth and then complain about shallow feedback. The format itself teaches closure manners. Time-boxed critique is how evidence review becomes a habit rather than a performance. Facilitators protect the action window as fiercely as the presentation slot. Rooms that honour the clock produce sharper comments in less wall time.
37.2 Checklist
Reviewers work from a checklist covering scope clarity, evidence hashes, severity rationale, privacy and overclaim. Freeform vibe reviews miss the same holes every term. Sam requires at least one note per checklist line even if the note is a clean pass. Overclaim catches are especially prized because they prevent board-level fiction. The checklist travels with the seminar note into the capstone file. Shared checklist language makes feedback comparable across presenters. Empty checklist lines are treated as incomplete reviews, not optional silence. Pass notes still prove the reviewer looked; silence does not.
37.3 Feedback
Specific line references; suggest fixes.
Useful feedback cites line numbers or figure ids and suggests a fix shape. Comments like tighten this teach nothing actionable before the next draft. Sam models one line-referenced fix in every seminar he facilitates. Presenters may ask clarifying questions but the default is to write actions quietly. Specificity is kindness under another name. If a comment cannot point to a place in the artefact, it is not ready to speak. Fix shapes can be rough as long as they point somewhere concrete in the draft. Presenters paste line anchors into the tracker so fixes navigate cleanly.
37.4 Receiving
Write actions; avoid debating every point live.
Receiving critique means writing actions, not winning the room. Live debate on every point burns the ten-minute action window. Sam's coaching is to mark agree, clarify later or decline with reason after the session. Ego defence during critique correlates with stale comment boards later. Quiet receiving is a graded professional skill. The seminar rewards people who leave with a clearer to-do list than when they entered. Later clarification threads beat courtroom energy in the seminar room. Action lists written in the room beat reconstructed lists the next morning.
37.5 Compare
Optionally compare to a published methodology checklist (PTES/WSTG), not to Instagram lore.
Optional comparisons use published methodology checklists such as PTES or WSTG excerpts provided in the course pack. Instagram lore and hallway myths are out of bounds as standards. Sam wants students to taste external yardsticks without worshipping any single brand. When a gap appears against the checklist, the action names which artefact will change. Lore comparisons are redirected to the pack. Published checklists keep critique anchored when opinions get loud. Course-pack excerpts keep everyone looking at the same external yardstick.
37.6 Record
Seminar note filed with capstone.
The seminar note is filed beside the capstone materials so examiners can see the critique trail. Notes include checklist marks, actions with owners and any declined comments with reasons. Sam rejects verbal-only seminars as incomplete education records. Filing happens the same day while memory is fresh. Missing notes block the capstone readiness checkbox. If it is not filed, the seminar did not count for the programme. Same-day filing stops the note from becoming a half-remembered myth. Readiness checkboxes that depend on filing keep incentives aligned.
37.7 Decline with reason
Seminar declined feedback requires a written reason the presenter accepts. "I disagree" without reason keeps the comment open. Sam's dossier once had six open comments; none remained after he wrote reasons or fixes.
Decline is allowed; unexplained decline is not closure. Writing the reason often reveals whether the comment was actually right. Sam's six-open story is told to show that clarity empties boards faster than stubbornness. Trackers keep declined-with-reason as a first-class state. Reasons turn disagreement into an auditable decision instead of a sulk. Accepted reasons are stored beside the comment so examiners can follow the choice.
37.8 Action expiry
Seminar actions older than fourteen days without state change page Sam. Stale actions are declined or done.
Expiry converts polite limbo into a ping that demands a state change. Sam would rather decline cleanly than pretend an action is still alive. Fourteen days matches the sprint-like rhythm of capstone work. Pagers feel rude until people notice how many threads they finish. Stale kindness is how comment boards become graveyards. A ping that forces done or declined is kinder than a forever-open card. Visible expiry dates remove the social awkwardness of nagging peers.
37.9 Comment state machine
Comment states are open, fixed, declined, risk-accepted. Sam's tracker forbids other strings.
Free-text statuses invent comforting lies like kinda done. The four allowed states cover honest outcomes without poetry. Risk-accepted requires an authority note the same way findings do. API validation rejects creative synonyms when people are tired. Pedantic state machines are a gift to future readers of the board. Synonym bans sound petty until they stop a midnight self-deception. Four states are enough for truth and few enough to learn cold.
37.10 Day-ten pager scene
Open seminar comments page Sam at day 10. He declines one with reason "out of scope for thesis" and fixes two. Day 14 board is clear. The pager teaches closure.
The scene is ordinary operations on a comment board. Day ten is early enough to act and late enough to feel pressure. Sam's out-of-scope decline is specific so the commenter can reuse the idea elsewhere. Clear boards by day fourteen are celebrated without fireworks. Pagers teach what posters only recommend. Early pressure leaves enough calendar to fix work before evaluation week. Clear boards are a quiet flex the programme is willing to name aloud.
37.11 No kinda-done status
State machine API rejects "kinda done". Sam is grateful for pedantic software when tired.
Kinda done is how work hides while looking progressive. Rejection at the API means the UI never offers the lie. Sam has tried to type it at midnight and been saved by the error. Fatigue is exactly when soft statuses appear. Software that refuses fuzzy closure protects the human who wants to go home. Error messages that name the forbidden string teach the rule in the moment. Midnight authors need fences more than lectures.
37.12 Deep links in pager messages
Pager messages include deep links to the comment thread.
A pager without a link creates a scavenger hunt through old channels. Deep links open the exact comment so the action can start in seconds. Sam forbids see thread shorthand in pager text. Broken links are ops defects, not presenter defects. Frictionless jumps from alert to artefact are how closure stays likely. Seconds matter when the alternative is losing the thread in scrollback. Link hygiene is comment hygiene with a URL.
37.13 Exact comment deep links
Deep links open the exact comment; Sam forbids "see thread" without links.
See thread is ambiguous when a channel moved or was renamed. Exact links remove arguments about which remark was meant. Sam's template pastes the URL automatically from the tracker. Reviewers who omit links have their pager messages bounced by a bot. Precision in pointers is part of comment hygiene. Bots that bounce incomplete pager text train better habits than memos do. Ambiguity about which remark was meant wastes the scarce action minutes.
37.14 Comment hygiene week
Comment hygiene week every term closes stale threads. Sam enjoys the clean board more than new features. Closure is a deliverable.
Hygiene week is scheduled like a release, not improvised guilt. Teams close, decline or risk-accept whatever remains. Sam treats a clean board as a product outcome worth mentioning in retros. New features wait if hygiene is behind. A culture that ships closure will also ship better tools. Retros that praise clean boards make hygiene socially profitable. Hygiene week on the calendar beats hygiene guilt in the hallway.
37.15 Open-count CSV pressure
Sam's seminar tracker exports a weekly CSV of open comments by presenter. Public (team-visible) open counts create gentle pressure without shame. Presenters with zero opens get a thank-you note. The social protocol closes more threads than any angry reminder ever did in prior terms.
Visibility without humiliation is a delicate design. Counts are by presenter open items, not personal attacks in chat. Thank-you notes make zero a celebrated state. Angry reminders had been tried; they created hiding, not closure. Social protocols can outperform scolding when the metric is fair and kind. Fair metrics plus kindness outperform fear as a closure engine.
37.16 Personal thank-you notes
Thank-you notes for zero opens are templated but personally sent.
Templates keep the bar consistent; personal sending keeps the thanks human. Sam writes one specific sentence about a closed thread so it does not feel automated. People forward those notes to themselves on hard weeks. The cost is minutes; the effect on board hygiene is outsized. Recognition that names real work beats generic badges from a bot. One specific sentence of thanks is remembered longer than a leaderboard. Humans sending thanks keep the ritual from rotting into spam.
Worked example (synthetic TRB)
Peer catches Sam's missing denominator on a coverage slide; Sam adds it before the CISO rehearsal.
Synthetic case — Riverstone Transit Bank (TRB). The bank, Sam Ito, TrackScope/FindingsDesk/PulseMap and all figures are invented for teaching.
Errors and limits
This book grants no authority to test real banks or payment systems. Labs use documentation-range addresses and synthetic data only; exploit code is never provided.
Exercises
- EX109 — Checklist item. Name four seminar checklist items. Rubric: 3 = four from chapter; 2 = three; 1 = one; 0 = none.
- EX110 — Feedback quality. What makes feedback actionable? Rubric: 3 = specific lines + suggested fix; 2 = partial; 1 = "bad vibes"; 0 = none.
- EX111 — Denominator. Why add denominators on coverage slides? Rubric: 3 = avoid misleading percentages; 2 = partial; 1 = decoration; 0 = none.
Synthesis
Seminars improve dossiers through specific, recorded critique.
Continue studying — chapter 37 of 40
← Chapter 36 (draft) · Course chapter index · Chapter 38 (draft) →