Chapter 23 — Authorized offensive security and red team: evaluation
Lesson map
Lesson 23 published. Map, example, EX067.
flowchart TD
N0["Learning outcomes"]
N1["23.1 Criteria before close"]
N2["23.2 Process quality ≠ trophy count"]
N3["23.3 Lessons learned"]
N4["23.4 Vanity metrics"]
N5["23.5 Sharing results without shame"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX067"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["23.1 Criteria before close"]
N2["23.2 Process quality ≠ trophy count"]
N3["23.3 Lessons learned"]
N4["23.4 Vanity metrics"]
N5["23.5 Sharing results without shame"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX067"]
[Learning outcomes]
|
v
[23.1 Criteria before close]
|
v
[23.2 Process quality ≠ trophy count]
|
v
[23.3 Lessons learned]
|
v
[23.4 Vanity metrics]
|
v
[23.5 Sharing results without shame]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX067]
Learning outcomes
You will (1) define evaluation criteria before an engagement ends; (2) score process quality separately from finding count; (3) run a lessons-learned review that produces backlog items; (4) refuse vanity metrics.
23.1 Criteria before close
Evaluation answers: did we stay in scope, did we meet the decision the CISO needed, did defenders learn, did remediation start? Write those criteria in the ROE or kickoff notes so future-you cannot invent softer ones after a messy week.
For ENG-712 the four questions were written into the kickoff notes as yes or no items: stayed in scope, supported the CISO's coverage decision, gave defenders something to learn, started remediation. Priya fills them in before the CISO reads the sheet, and the finding count appears only as a supporting line, so a long appendix cannot stand in for a missed criterion.
23.2 Process quality ≠ trophy count
An engagement that finds nothing but demonstrates solid gates, clean evidence and a useful purple replay can be a success. An engagement that finds dozens of issues by ignoring exclusions is a failure with a long appendix. Score discipline explicitly.
Priya's studio rubric puts this on its first page, because students tend to measure themselves by the length of the findings table. In ENG-712 a quiet engagement with clean gates and a useful purple replay would still have met all four criteria. A team that produced many findings by crossing a STOP line would fail the scope criterion, and no finding count can repair that.
23.3 Lessons learned
Within two weeks, red, blue and white teams meet. Each miss from the coverage matrix becomes a ticket. Each process fault becomes a procedure change. Capture what to stop, start and continue.
DET-19 from the coverage story is the model miss ticket: it names the uncovered ATT&CK technique, the matrix cell it came from and the detection owner. Process faults are written as edits to a procedure, such as the clock-sync check before kickoff that ENG-712 added. Holding the meeting within two weeks matters because people still remember why a choice was made.
23.4 Vanity metrics
"Hosts scanned", "packets sent" and "criticals found" without context encourage the wrong behaviour. Prefer "techniques exercised", "percent alerted", "findings accepted into tracker within SLA" and "retests closed".
On a management slide Priya replaces a row such as hosts scanned with the number of techniques exercised and the share that raised an alert. The second pair answers the question the CISO asked about SOC coverage; the first only measures how busy the testers were. She removes vanity rows in the same review pass that strips forbidden assurance phrases from the report.
23.5 Sharing results without shame
Publish the evaluation sheet to the people who lived the exercise. Hiding a low process score from the SOC while celebrating a high finding count teaches the wrong lesson. TRT's rule is that the same sheet the CISO signs is the sheet the team reads in the retrospective.
In ENG-712 the SOC saw the full sheet, including the clock-sync process fault, at the same retrospective where the two detection backlog items were discussed. Hearing about a weak process score from a summary weeks later would have felt like blame. Seeing it beside the findings made it a shared problem with an owner and a date.
Worked example
TRT's evaluation sheet for ENG-712: scope adherence 100% (no STOP breaches); decision supported (SOC coverage numbers delivered); two detection backlog items opened; one process change (clock sync check before kickoff). Finding count is recorded but not used as the primary score. The CISO signs the sheet.
Synthetic case. The evaluation sheet is invented.
Errors and limits
Blaming individuals in the lessons-learned meeting destroys purple-team trust. Keep the minutes about systems and procedures.
Exercises
- EX067 — Primary score. Why is "criticals found" a weak primary evaluation score? Rubric: 3 = rewards volume/scope abuse; ignores process and learning; 2 = one reason; 1 = it is the best score; 0 = none.
- EX068 — Criterion. Write one pre-declared criterion that measures defender learning. Rubric: 3 = measurable learning criterion (e.g. backlog items opened per miss, replay attendance); 2 = vague learning; 1 = finding count; 0 = none.
- EX069 — Stop/start/continue. Give one stop, one start and one continue from ENG-712's story so far. Rubric: 3 = three concrete items grounded in prior chapters; 2 = two; 1 = one; 0 = none.
Synthesis
Declare criteria early, score discipline, convert lessons into tickets, and ignore vanity counters.
Continue studying — chapter 23 of 30
← Chapter 22 (draft) · Course chapter index · Chapter 24 (draft) →