Chapter 27 — Verification
Lesson map
Lesson 27 published. Map, example, EX079.
flowchart TD
N0["Learning outcomes"]
N1["27.1 Retest"]
N2["27.2 Coverage lab"]
N3["27.3 Closure"]
N4["27.4 Partial"]
N5["27.5 Scope freeze"]
N6["27.6 Communicate"]
N7["27.7 GAP stories for SOC"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX079"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["27.1 Retest"]
N2["27.2 Coverage lab"]
N3["27.3 Closure"]
N4["27.4 Partial"]
N5["27.5 Scope freeze"]
N6["27.6 Communicate"]
N7["27.7 GAP stories for SOC"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX079"]
[Learning outcomes]
|
v
[27.1 Retest]
|
v
[27.2 Coverage lab]
|
v
[27.3 Closure]
|
v
[27.4 Partial]
|
v
[27.5 Scope freeze]
|
v
[27.6 Communicate]
|
v
[27.7 GAP stories for SOC]
|
v
[Worked example] --> [Exercise EX079]
Learning outcomes
You will (1) plan retests; (2) read coverage labs; (3) close only with evidence.
27.1 Retest
Same methods, agreed window.
A retest repeats the original proof steps inside a window the owner and the tester both accepted when the finding was filed. Changing the method mid-retest makes a FIXED claim dishonest, because the owner may have patched only the path that was demonstrated. Sam books the window when the finding lands, usually about two weeks out, so the calendar does not wait for panic. If the build under test has drifted, the retest records that drift instead of inventing a pass. The point of the exercise is truth about closure, not a green checkbox.
27.2 Coverage lab
lab_coverage flags T1190 GAP.
"""Lab 27: summarise synthetic purple-team results per ATT&CK technique."""
runs = [("T1078",True),("T1078",True),("T1078",False),("T1566",True),("T1566",False),("T1190",False),("T1190",False)]
from collections import defaultdict
d = defaultdict(lambda:[0,0])
for t, hit in runs: d[t][1]+=1; d[t][0]+=hit
for t in sorted(d): k,n=d[t]; print(f"{t} detected {k}/{n} {'GAP' if k/n<0.5 else 'ok'}")
Run with python3 lab_coverage.py. Verified output:
T1078 detected 2/3 ok
T1190 detected 0/2 GAP
T1566 detected 1/2 ok
The lab walks a short list of synthetic purple-team runs and prints how often each technique was detected. Learners read the printed ratios before they argue about tooling. A technique that never hits is a gap story, not a reason to declare the SOC broken. Sam's coaching focuses on framing: small samples can still justify instrumentation when the log source already exists. Students who only celebrate high hit rates miss the engineering value of an honest miss.
27.3 Closure
FIXED with evidence or accepted risk signed.
Closure is either a FIXED status backed by a retest transcript or an accepted-risk signature from someone with authority to own the residual. Anything else leaves the ticket open. Sam refuses to mark FIXED from a verbal "we patched it" without the same request that proved the issue. Accepted risk is not a soft fail; it is a documented decision with an expiry and an owner. FindingsDesk will not let a ticket disappear into a vague "monitoring" state without one of those two endings.
27.4 Partial
STILL OPEN keeps ticket alive.
When a retest shows the path is narrowed but not closed, the status stays STILL OPEN and the ticket remains on the owner's queue. Partial progress is recorded in the notes so nobody pretends the work never happened, but progress is not closure. Sam once saw a team celebrate a WAF rule that blocked his old payload while a one-character variant still worked; the ticket stayed open and the celebration moved to the next patch. Honesty about partial states keeps leadership from declaring victory on a dashboard that lies.
27.5 Scope freeze
No new hunt on retest day.
Retest day is for verifying known findings, not for opening a fresh hunt. New leads go into a parking note for the next engagement window. The freeze protects the owner's change window and keeps the evidence file coherent. Sam has watched retests turn into sprawling discovery days that produced neither a clean FIXED nor a clean STILL OPEN. The rule is posted on the retest calendar invite so nobody claims they were surprised. If a critical new lead appears anyway, it becomes a separate emergency ticket with its own approvals rather than hitchhiking on the retest.
27.6 Communicate
Same-day owner note.
After a retest, the owner receives a same-day note that states FIXED, STILL OPEN or accepted-risk pending, with a link to the transcript. Waiting until the weekly report creates rumours and duplicate work. Sam writes the note before he shuts the laptop, even when the news is disappointing. The note names the method card hash so the owner can confirm the same build was tested. Speed here is courtesy, not pressure. A late note forces the owner to ask peers what happened, which is how incorrect status stories start.
27.7 GAP stories for SOC
GAP stories offered to SOC include log source, technique and a sample detection idea — not a demand to "buy a tool". Sam's T1190 note pointed at WAF and origin logs already licensed.
A useful gap story tells the SOC what to look at with what they already own. Sam's T1190 write-up named the WAF bypass logs and the origin access logs, both already licensed, and sketched a detection idea that used those fields. Demanding a new product in a gap note usually stalls the conversation. Detection engineers respond faster when the ask is concrete and cheap. The backlog item that followed cited Sam's note verbatim.
27.8 Retest calendar
Retest windows are booked when findings file, not after owners panic. Sam's template suggests a date 14 days out.
Booking early turns retest from a crisis into a routine. The template proposes a fourteen-day offset, which gives owners time to patch without inventing urgency theatre. If the owner needs longer, they say so in the ticket and the calendar moves with a reason attached. Sam's admin once tried to stack five retests on one Friday; the calendar view made the overload obvious before anyone burned out. Predictable windows are how verification stays kind.
27.9 GAP to backlog
GAP rows create detection backlog items with technique ids. PulseMap links to those items.
Every GAP row in PulseMap must point at a detection backlog item that carries the same technique identifier. The link is bidirectional so grooming can find orphans in either direction. Sam files a platform defect when a GAP sits for a sprint without a backlog id. The defect is about process, not about blaming the analyst who filed the gap. Linking keeps purple-team output from becoming a graveyard of screenshots.
27.10 T1190 gap becomes a spike
T1190 GAP with n=2 becomes a detection spike story: "instrument WAF bypass logs". Two weeks later SOC has a rule in staging. Small n still can drive engineering if framed as a gap, not a doom claim.
Two misses would look weak in a slide about coverage percentages. Framed as a gap with a named log source, they became a two-week instrumentation spike that landed a rule in staging. Sam's write-up avoided words like "blind" and "critical failure" and stuck to what was missing. The SOC lead later said the tone is why the story moved. Evidence size and evidence usefulness are different questions.
27.11 Method hash in invites
Retest calendar invites include the method card hash so owners prepare the same build. Version skew is the enemy of honest FIXED.
Owners sometimes patch a different build than the one Sam tested. Putting an eight-character SHA-256 prefix in the invite forces the mismatch into the open before the session starts. If the hashes differ, the retest either retargets or records the skew explicitly. Sam would rather reschedule than stamp FIXED on a cousin of the vulnerable build. Version honesty is part of verification ethics.
27.12 Bidirectional GAP links
Detection backlog items link both ways to PulseMap GAP rows.
A one-way link dies quietly when someone renames a ticket. Bidirectional links let either side notice a break during grooming. Sam's weekly check is a five-minute query for GAP rows without backlog ids and backlog items whose PulseMap link 404s. Fixing those breaks is dull and necessary. Purple-team value is mostly in the plumbing that keeps signals attached to work. When both directions resolve, a detection engineer can walk from a miss to a ticket without asking Sam where the screenshot went.
27.13 Hash prefixes in invites
Method card hashes in invites are SHA-256 prefixes of eight hex chars — enough to spot skew.
Eight hex characters are short enough to paste into a calendar field and long enough to make accidental collisions unlikely inside one engagement. Full digests live in the notebook for anyone who wants to recompute. Sam once caught a skew because the prefix in the invite no longer matched the card on the share. The owner had rebuilt overnight and forgotten to say so. Short prefixes earn their keep when they prevent a false FIXED.
27.14 Backlog grooming with PulseMap
Detection backlog grooming includes PulseMap once a sprint. GAP rows without backlog ids are defects in the grooming, not in the SOC. Sam files those defects against the platform process.
Grooming that ignores PulseMap slowly invents a shadow backlog of screenshots. Including the GAP view once a sprint keeps the two lists honest. When Sam finds an orphan GAP, he files against the process that should have linked it, not against the analyst who was busy that week. Process defects are cheaper to fix than cultural blame. The sprint ends with every open GAP pointing somewhere real.
Worked example (synthetic TRB)
PulseMap-style lab shows T1190 0/2 GAP; SOC prioritises public-facing detection engineering.
Synthetic case — Riverstone Transit Bank (TRB). The bank, Sam Ito, TrackScope/FindingsDesk/PulseMap and all figures are invented for teaching.
Errors and limits
This book grants no authority to test real banks or payment systems. Labs use documentation-range addresses and synthetic data only; exploit code is never provided.
Exercises
- EX079 — GAP line. What does the lab print for T1190? Rubric: 3 = detected 0/2 GAP; 2 = partial; 1 = ok; 0 = none.
- EX080 — T1078. Is T1078 GAP or ok in the lab? Rubric: 3 = ok (2/3); 2 = GAP; 1 = missing; 0 = none.
- EX081 — Closure rule. When may a finding close as FIXED? Rubric: 3 = retest evidence shows closed; 2 = partial; 1 = owner email only; 0 = none.
Synthesis
Verification mixes retest honesty with detection coverage gaps.
Continue studying — chapter 27 of 40
← Chapter 26 (draft) · Course chapter index · Chapter 28 (draft) →