Chapter 38 — Authorized offensive security and red team: validation
Lesson map
Lesson 38 published. Map, example, EX112.
flowchart TD
N0["Learning outcomes"]
N1["38.1 Re-test design"]
N2["38.2 Negative results"]
N3["38.3 No new methods"]
N4["38.2b Sampling re-tests"]
N5["38.3b Automating re-test safely"]
N6["38.1b Evidence of validation"]
N7["Practice note"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX112"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["38.1 Re-test design"]
N2["38.2 Negative results"]
N3["38.3 No new methods"]
N4["38.2b Sampling re-tests"]
N5["38.3b Automating re-test safely"]
N6["38.1b Evidence of validation"]
N7["Practice note"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX112"]
[Learning outcomes]
|
v
[38.1 Re-test design]
|
v
[38.2 Negative results]
|
v
[38.3 No new methods]
|
v
[38.2b Sampling re-tests]
|
v
[38.3b Automating re-test safely]
|
v
[38.1b Evidence of validation]
|
v
[Practice note]
|
v
[Worked example] --> [Exercise EX112]
Learning outcomes
You will (1) validate fixes with re-tests; (2) avoid validation that reintroduces risk; (3) record negative results.
38.1 Re-test design
Same vantage, least privilege, same evidence standard. For default password: confirm login fails; do not spray variants.
Spraying password variants turns validation into a new attack. Confirming the default login fails, from the same vantage and privilege, is enough to close SYN-001. Sam's re-test card names the exact request and the expected 401, nothing more. A request to try alternate passwords on the same form is recorded as a refused method, not as thoroughness. The refusal is visible on the ticket so a later auditor sees the boundary was held.
38.2 Negative results
"Still open" is a valid validation outcome. PulseMap and FindingsDesk must accept it without forcing close.
Forcing every ticket closed trains people to lie when a fix failed. "Still open" with fresh evidence is how SYN-005 stayed honest until the group membership actually changed. FindingsDesk accepts that status without a close reason, and PulseMap counts it as incomplete validation, not as success. Closing SYN-005 required a second re-test after the group membership finally disappeared, with a new evidence hash. The SLA clock did not reset on the failed first re-test; it kept running until the second green.
38.3 No new methods
Validation uses the method that found the issue, or a weaker read-only check — not a chance to try new attacks.
Synthetic case — Riverstone Transit. Re-test of SYN-001: default login returned 401; ticket closed with evidence hash. SYN-005 still had group membership — left open.
Validation is not a second engagement. Using a weaker read-only check, or the same method that found the issue, keeps the ROE stable. Sam rejected a proposal to "just check one more port" during the SYN-001 re-test; that port belonged on the next engagement's plan if the client wanted it.
38.4 Errors and limits
Different environment (staging vs prod) means you validated staging.
Closing a production finding because staging looks fixed is how breaches return after go-live. The re-test record must name the environment. Sam's ticket template refuses a close when the environment field still says staging and the finding's asset is production. Staging greens may be attached as supporting notes but never as the sole close evidence. A production re-test window is booked on the same calendar as discovery so it cannot be forgotten.
38.2b Sampling re-tests
When many similar findings share a root cause (fleet-wide header missing), re-test a sample plus the originally reported hosts, and say so. Claiming "all 200 closed" after checking five is a measurement lie.
Re-testing every low finding can burn the window; saying you sampled, and how, keeps the claim honest. Stratify by severity so criticals are never in the unsampled remainder. Sam's validation report for a fifty-finding engagement listed the ten sampled lows and the rule that picked them. Critical and high findings were re-tested at 100% in that engagement; the sample applied only to lows.
38.3b Automating re-test safely
Automated re-tests are allow-listed checks (HTTP status, group membership read) with hard timeouts. They never include credential stuffing or exploit retries. If automation cannot validate safely, schedule a human read-only check.
Automation that replays the original exploit path is out of bounds for this programme. Safe automation means authorized reads and expected status codes, like the 401 check on SYN-001. Sam's automation library will not load a method card marked noisy for validation. Cards marked quiet may automate; anything that needs a rate-limit exception stays manual.
38.1b Evidence of validation
Validation tickets link to the original finding, the method used, the timestamp, and the hash of the new evidence. "Looks good to me" without those fields does not close a critical.
A comment that says "looks fixed" is not evidence. Hashes of the new response, time, vantage and account belong on the ticket the same way discovery evidence did. Sam closes only when those fields are present and the hash matches the sealed store. A close comment without a hash is rejected by FindingsDesk the same way an empty severity is.
Practice note
Disputes over validation escalate to a named adjudicator role with access to both evidence hashes. The adjudicator does not re-attack; they compare evidence to the acceptance note. Document the decision in the ticket so the argument does not restart at every status meeting.
Require the re-test method id on every transition to closed for critical and high severities.
38.4b Communicating a failed re-test
When a re-test shows a fix did not work, the report says so plainly, includes the evidence and the date, and avoids suggesting blame. The Riverstone Transit owner receives it privately first, so the team can plan a new fix before the result reaches wider governance meetings.
Owners deserve a clear "still open" with the new evidence, not silence until the retro. The message reuses the finding id and avoids new attack detail. Sam's template for SYN-005 named the still-present group membership and the SLA clock still running, then offered a re-test slot after the next change.
Exercises
- EX112 — Re-test. Steps to validate default-password fix safely. Rubric: 3 = single attempt, expect fail, evidence, no spray; 2 = partial; 1 = spray; 0 = exploit.
- EX113 — Negative. Why must "still open" be storable? Rubric: 3 = honesty/SLA; 2 = vague; 1 = always close; 0 = none.
- EX114 — Env. One sentence limitation when re-test was staging-only. Rubric: 3 = clear; 2 = vague; 1 = claims prod; 0 = none.
Synthesis
Validate narrowly; keep negatives; don't widen methods.
Continue studying — chapter 38 of 50
← Chapter 37 (draft) · Course chapter index · Chapter 39 (draft) →