Chapter 34 — Comparative project
Lesson map
Lesson 34 published. Map, example, EX100.
flowchart TD
N0["Learning outcomes"]
N1["34.1 Options"]
N2["34.2 Criteria"]
N3["34.3 Evidence"]
N4["34.4 Recommendation"]
N5["34.5 Non-goals"]
N6["34.6 Presentation"]
N7["34.7 Capacity to execute row"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX100"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["34.1 Options"]
N2["34.2 Criteria"]
N3["34.3 Evidence"]
N4["34.4 Recommendation"]
N5["34.5 Non-goals"]
N6["34.6 Presentation"]
N7["34.7 Capacity to execute row"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX100"]
[Learning outcomes]
|
v
[34.1 Options]
|
v
[34.2 Criteria]
|
v
[34.3 Evidence]
|
v
[34.4 Recommendation]
|
v
[34.5 Non-goals]
|
v
[34.6 Presentation]
|
v
[34.7 Capacity to execute row]
|
v
[Worked example] --> [Exercise EX100]
Learning outcomes
You will (1) compare two assessment approaches; (2) use TRB criteria; (3) recommend a mix.
34.1 Options
Quarterly WSTG-style web test vs annual threat-led scenario exercise.
The comparative project sets two real options against each other rather than against a fantasy of infinite budget. Quarterly web tests following a WSTG-style method cover common application flaws on a steady rhythm. An annual threat-led scenario digs deeper into adversary paths but consumes far more hours. Sam refuses a third option that claims to do both at full intensity every quarter. Honest options make the later recommendation defensible. Choosing among feasible programmes beats designing an unfunded masterpiece.
34.2 Criteria
Cost, coverage, detection learning, operational risk, examiner expectations.
Criteria are fixed before scores are filled so the answer cannot steer the rubric. Cost includes staff hours and vendor invoices; coverage asks what asset classes are touched; detection learning asks what the SOC gains; operational risk asks what could break; examiner expectations ask what supervisors will look for. Sam weights the criteria with the sponsor before scoring begins. Hidden criteria that appear in the recommendation paragraph are a fail. A public rubric is how comparative work earns the right to advise spending.
34.3 Evidence
Use synthetic past results and Wilson intervals.
Evidence for the comparison comes from synthetic past results with Wilson intervals around detection and coverage rates. Point estimates without intervals make small samples look decisive. Sam's tables show the interval beside every rate the slide will cite. Synthetic history keeps the exercise shareable while still punishing sloppy stats. Students who drop the intervals to tidy a chart lose marks for evidence hygiene. Interval literacy belongs in programme design, not only in research electives.
34.4 Recommendation
The recommended mix keeps quarterly web tests for steady coverage and funds a threat-led scenario every other year. Annual threat-led scored well on learning and poorly on capacity; the biennial cadence won on the capacity row. Sam states what the mix intentionally leaves on the table so sponsors cannot invent full annual depth later. Recommendations name the cut, not only the prize. A mix that admits limits survives contact with the budget committee. Sam attaches the scored option table as an appendix, so the committee can check every row behind the recommendation before it votes.
34.5 Non-goals
Not recommending zero testing.
One explicit non-goal is recommending zero testing as a savings measure. Another is pretending vendor marketing slides are evidence. Sam prints non-goals beside the options so the debate cannot quietly redefine success. Comparative projects that skip non-goals drift into whichever option the loudest voice likes. Clear refusals keep the recommendation honest when pressure rises. A third non-goal is scoring options on prestige rather than on the published criteria. Saying what you will not recommend is part of recommending well.
34.6 Presentation
Ten slides, decisions last.
The presentation is capped at ten slides with the decision on the final one. Front-loading theatre and hiding the ask is graded as a miss. Sam's coaching is to put criteria and capacity arithmetic before adjectives. Executives who only see the last slide still deserve a self-explanatory decision box. Appendix material stays in the appendix. Speaker notes may be rich; the visible deck stays lean. Short decks force the comparison to carry the weight instead of the speaker's charm.
34.7 Capacity to execute row
Comparison tables include a "capacity to execute" row. Threat-led every year scored high on learning and low on capacity; every other year won. Ignoring capacity produces shelfware programmes.
Learning scores seduce sponsors; capacity scores keep programmes real. Sam's table made the annual threat-led option look excellent until the capacity row turned red. Shelfware programmes are how banks buy prestige and ship nothing. The row uses last year's hours, not optimism. When capacity is ignored, the recommendation becomes fiction. Execution capacity is a criterion with the same dignity as coverage.
34.8 Cost honesty
Comparison tables include staff hours, not only vendor invoices. Internal quarterly tests are not free.
Invoice-only costing makes internal work look free and biases the table toward busywork the staff already cannot finish. Sam converts hours to cost with a published internal rate so apples match apples. Hidden overtime is still cost even when finance has not booked it. Students who omit hours fail the honesty check on the rubric. Sponsors deserve the full price of the mix. Cost honesty prevents comparative theatre from laundering unpaid labour.
34.9 Capacity row data
Capacity estimates come from last year's hours, not optimism. Sam cites the hours file.
Optimism is not a dataset. Sam attaches the time-tracking export hash beside the capacity row so reviewers can recompute. Handmade estimates once undercounted threat-led work by roughly forty percent. Citing the file turns a debate about vibes into a debate about arithmetic. Future years update the file rather than inventing a new folklore number. Evidence-backed capacity rows are how mixes stay fundable after leadership changes.
If the export shows a spike from a one-off incident, Sam footnotes it rather than smoothing it away, so the board sees why the average looks heavy.
34.10 Hours arithmetic scene
Capacity row: last year threat-led consumed 1,200 hours. Quarterly web tests consumed 400. The mix recommendation falls out of the arithmetic more than out of fashion.
The scene is a quiet spreadsheet moment rather than a dramatic pitch. Twelve hundred hours beside four hundred makes the biennial threat-led choice feel inevitable. Sam lets the numbers speak before he narrates them. Fashionable frameworks lose to arithmetic when the room is tired. Students practice reading the row aloud without apology. When the arithmetic is solid, the recommendation needs less theatre to survive. Graders look for the refusal as carefully as they look for the recommendation in this comparative cycle.
34.11 Time-tracking export
Hours file is exported from time tracking, not remembered. Sam's first handmade estimate was 40% low. Humility is a spreadsheet.
Memory compresses painful projects and inflates tidy ones. The export ends arguments about whether last year was really that heavy. Sam keeps the undercount story in the facilitator guide as a humility prompt. Hashes on the export stop quiet edits after the meeting. Comparative work inherits the same evidence manners as findings work. Spreadsheets that embarrass the author are still better than confident fiction.
34.12 Hours export in appendix
Hours export is attached to the comparative project appendix as evidence.
Appendix evidence lets a skeptical board member recompute without calling Sam at night. The export is stripped of personal names where policy requires and keeps role labels. Missing exports make the capacity row look decorative. Sam's checklist for the project packet lists the export beside the slide deck. Attachments are how arithmetic becomes auditable. A recommendation without its hours appendix is an opinion in nice fonts.
The appendix also states which time-tracking codes were counted as testing, because reporting and retest hours are easy to leave out by accident.
34.13 Refuse handmade hour estimates
Comparative appendices refuse handmade hour estimates without a time-tracking export hash.
The refusal is mechanical: no hash, no capacity claim. Sam would rather delay a board date than bless a guessed row. Handmade numbers return through the side door whenever the rule is soft. Students learn to request exports early because late requests slip. Harshness here protects everyone from a programme that cannot be staffed. Guarding the capacity row is guarding the recommendation's integrity. Finance questions get sharper when the page still makes sense without the speaker in this comparative cycle.
34.14 What would you cut first
Comparative project defences ask "what would you cut first if budget halves?". Sam prepares that answer before the nice answer. Capacity thinking is the point.
The cut question reveals whether the student understood capacity or only memorised a mix slogan. Sam answers with which quarter of web testing shrinks and which threat-led elements wait. Preparing the painful answer first keeps the nice answer from sounding naive. Sponsors trust advisors who can retreat in good order. Capacity thinking is the real exam hiding inside the project. Knowing what to cut is part of knowing what to fund.
34.15 Two-year calendar visual
When presenting the mix recommendation, Sam brings a one-page calendar for two years showing quarterly web tests and a shaded threat-led quarter. Visual capacity beats abstract ratios. Executives approve calendars more readily than adjectives like "robust programme". He leaves copies of the calendar in the board portal the same day.
A calendar makes collisions with freezes and examiner visits obvious. Shading the threat-led quarter prevents silent double booking of the same staff. Abstract ratios invite head-nods; calendars invite real questions. Same-day portal upload stops hallway versions from drifting. Visual artefacts travel when speakers cannot. Calendars convert programme poetry into something finance can schedule.
34.16 Calendar PDF for finance
Two-year calendar PDF is the artefact boards forward to finance.
Finance rarely forwards slide decks with metaphors intact. The one-page calendar PDF survives forwarding chains and still shows when money and people are needed. Sam versions the PDF with a date in the footer so stale copies are obvious. If finance asks for a third year, the model extends without rewriting the story. Artefacts designed for forwarding are artefacts designed for decisions. The forwarded calendar is often the only page that still matters a month later.
Worked example (synthetic TRB)
Sam's comparison table scores detection learning higher for threat-led and regression coverage higher for quarterly WSTG.
Synthetic case — Riverstone Transit Bank (TRB). The bank, Sam Ito, TrackScope/FindingsDesk/PulseMap and all figures are invented for teaching.
Errors and limits
This book grants no authority to test real banks or payment systems. Labs use documentation-range addresses and synthetic data only; exploit code is never provided.
Exercises
- EX100 — Higher regression. Which approach better catches web regressions each quarter? Rubric: 3 = quarterly WSTG-style; 2 = threat-led only; 1 = neither; 0 = none.
- EX101 — Detection learning. Which tends to teach SOC more about realistic paths? Rubric: 3 = threat-led scenario exercises; 2 = banner scans; 1 = neither; 0 = none.
- EX102 — Mix. What mix does Sam recommend? Rubric: 3 = quarterly web + threat-led every other year; 2 = partial; 1 = zero testing; 0 = none.
Synthesis
Comparison uses explicit criteria; the answer is usually a mix, not a silver bullet.
Continue studying — chapter 34 of 40
← Chapter 33 (draft) · Course chapter index · Chapter 35 (draft) →