Chapter 11 — Verification
Lesson map
Lesson 11 published. Map, example, EX041.
flowchart TD
N0["Learning outcomes"]
N1["11.1 Retest plan"]
N2["11.2 Before/after"]
N3["11.3 Partial fixes"]
N4["11.4 Closure"]
N5["11.5 Scope freeze"]
N6["11.6 Communicate"]
N7["11.7 Same method"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX041"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["11.1 Retest plan"]
N2["11.2 Before/after"]
N3["11.3 Partial fixes"]
N4["11.4 Closure"]
N5["11.5 Scope freeze"]
N6["11.6 Communicate"]
N7["11.7 Same method"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX041"]
[Learning outcomes]
|
v
[11.1 Retest plan]
|
v
[11.2 Before/after]
|
v
[11.3 Partial fixes]
|
v
[11.4 Closure]
|
v
[11.5 Scope freeze]
|
v
[11.6 Communicate]
|
v
[11.7 Same method]
|
v
[Worked example] --> [Exercise EX041]
Learning outcomes
You will (1) plan a retest window; (2) compare before/after statuses; (3) handle partial fixes; (4) close only with evidence.
11.1 Retest plan
List findings, methods, owners' go-live notes, window.
Retest plan for ENG-512 uses the same method cards and pinned tool versions. lab_retest-style status shows F1 open->closed FIXED, F2 open->open STILL OPEN, F3 open->closed FIXED — fixed: 2 open: 1.
Using the same cards and tool versions is what makes the split meaningful: if F2 is still open, the cause is the fix and not a change in how Sam looked. The notebook places the two-fixed, one-open summary beside the digests of the pinned tools, so an examiner can see that the retest and the original test used identical inputs.
11.2 Before/after
Record status transitions explicitly.
"""Lab 11: compare synthetic before/after findings for a retest window."""
before = {"F1": "open", "F2": "open", "F3": "open"}
after = {"F1": "closed", "F2": "open", "F3": "closed"}
for fid in sorted(before):
b, a = before[fid], after[fid]
print(f"{fid} {b}->{a} {'FIXED' if b == 'open' and a == 'closed' else 'STILL OPEN' if a == 'open' else 'CHECK'}")
print("fixed:", sum(1 for f in before if before[f] == "open" and after[f] == "closed"),
"open:", sum(1 for f in after if after[f] == "open"))
Run with python3 lab_retest.py. Verified output:
F1 open->closed FIXED
F2 open->open STILL OPEN
F3 open->closed FIXED
fixed: 2 open: 1
Before/after evidence hashes sit on the ticket and prove the artefact changed with the fix. FIXED without a MATCH hash on the after-artefact is not closure at NMT.
Each transition is written with its date and the evidence that justified it, for example F1 moving from open to closed FIXED with an after-artefact whose digest matches the vaulted copy. A status that changes without a recorded reason cannot be audited later, and NMT's examiners sample exactly these rows when they review closed tickets.
11.3 Partial fixes
F2 still open means the ticket stays open with new evidence.
Partial fixes that leave F2 STILL OPEN stay on the tracker with an owner and due date. Sam does not close the engagement as clean while open items remain.
The supervisor review checks that the summary slide shows F2 as STILL OPEN with its owner and due date rather than hiding it under a clean headline. Wording such as partly fixed is not used; the ticket either meets the check on its method card or it does not, and the new evidence explains which part of the fix is missing.
11.4 Closure
Closed means FIXED with evidence, or accepted risk with CISO signature — not silence.
Closure requires owner sign-off and verified status FIXED with evidence. Verbal done in chat is rejected in the weekly review.
Risk acceptance is the only other route, and it needs the CISO's signature on the ticket rather than an agreement in a meeting. The weekly review reopens any item that was closed on enthusiasm without vaulted evidence. The pair the review accepts is a stored hash of the after-artefact plus the owner's sign-off.
11.5 Scope freeze
No new hunt during retest unless authorized.
During retest the scope is frozen: no new hosts are added to ENG-512, even when a tool suggests another listener. The retest statuses therefore cover only listeners from the original list. The freeze text is read aloud at retest kickoff, and any newly suggested host is written into a parking list as a candidate for the next ROE, where owners can decide whether to authorise it. Retesting a new host without that decision would produce a result nobody had agreed to receive.
11.6 Communicate
Send a short retest report to owners the same day.
Communicate retest results to the SOC and owners the same day. STILL OPEN on F2 is not buried until the monthly pack.
Sam sends one message to the SOC lead and the finding owner together, so both hear about F2 at the same moment and neither learns it second-hand. The message timestamps are kept beside the ticket for exam packs. Waiting for the monthly pack would leave the owner a month behind on a weakness that is still open.
11.7 Same method
Retest uses the same method card, not a stronger one. Switching methods mid-retest confuses whether the fix worked.
Same method on retest: if F1 was a cookie flag check, retest uses that card, not a different scanner profile that invents a new pass.
If the original check used the cookie card, a different scanner profile might simply not test the same flag and report a pass that means nothing. A peer initials the ticket to confirm that the method card on the retest matches the one used for the finding. Changing the method is a new test, and it needs its own card and ROE line.
11.8 Owner demo
When helpful, Sam invites the owner to watch the successful retest once — seeing FIXED builds trust more than a ticket comment.
Owner demo on staging shows the FIXED state before Sam marks closure. Screenshots alone without the owner present are weak for examiner packs.
The owner sees the FIXED state on staging with their own eyes, which makes later disputes about whether the fix was real much less likely. The calendar invitation for the demo is filed as evidence of who witnessed it. Screenshots still go into the vault, but they support the demo rather than replace it.
11.9 Field discipline
Retest day is emotionally harder than find day because owners hope for green. Sam opens the call by restating that STILL OPEN is a successful retest outcome when true. He shows the before and after evidence side by side. When F2 remains open, he helps the owner schedule the next fix rather than arguing about severity again — severity was already accepted when the finding was filed.
Field discipline for verification is same method, hashed before/after, owner demo, and honest STILL OPEN counts. Two fixed and one open is the ENG-512 teaching result.
11.10 Closing note
If all findings FIX on first retest, Sam still records the methods and hashes. Easy wins deserve the same evidence trail as stubborn ones, especially when examiners sample closed tickets.
The closing note for ENG-512 makes the same point: verification is how NMT comes to trust the programme. Sam's reputation rests on reporting honest open counts, including F2, rather than producing a zero-open slide. A programme that pushes testers towards forced zero-open stories has a defect of its own, and supervisors are expected to name it.
Worked example (synthetic NMT)
Sam's retest: F1 and F3 FIXED, F2 STILL OPEN. He closes T-F1 and T-F3 with evidence, leaves T-F2 open, and notifies the owner.
Synthetic case — Northline Municipal Transit (NMT). The transit agency, Sam Park and all figures are invented for teaching.
Errors and limits
Live transit systems and fare equipment are never tested without a separate, specialised authorization. This book uses synthetic data only and grants no authority to test real systems.
Exercises
- EX041 — Lab counts. How many fixed and how many still open? Rubric: 3 = fixed 2 open 1; 2 = partial; 1 = wrong; 0 = none.
- EX042 — F2 action. What happens to the F2 ticket? Rubric: 3 = stays open with new evidence; 2 = open; 1 = close anyway; 0 = none.
- EX043 — Accepted risk. Who must sign if NMT accepts F2 without fix? Rubric: 3 = CISO (risk owner); 2 = partial; 1 = Sam; 0 = none.
- EX044 — No new hunt. Why freeze scope during retest? Rubric: 3 = retest measures fixes; new hunt needs new auth; 2 = partial; 1 = always hunt; 0 = none.
Synthesis
Verification is before/after evidence, honest open tickets and same-day communication.
Continue studying — chapter 11 of 18
← Chapter 10 (draft) · Course chapter index · Chapter 12 (draft) →