Chapter 30 — Verification laboratory I
Lesson map
Lesson 30 published. Map, example, EX088.
flowchart TD
N0["Learning outcomes"]
N1["30.1 Plan"]
N2["30.2 Execution"]
N3["30.3 Report"]
N4["30.2b Golden fixtures"]
N5["30.3b Independent runner"]
N6["30.1b Lab data retention"]
N7["Practice note"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX088"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["30.1 Plan"]
N2["30.2 Execution"]
N3["30.3 Report"]
N4["30.2b Golden fixtures"]
N5["30.3b Independent runner"]
N6["30.1b Lab data retention"]
N7["Practice note"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX088"]
[Learning outcomes]
|
v
[30.1 Plan]
|
v
[30.2 Execution]
|
v
[30.3 Report]
|
v
[30.2b Golden fixtures]
|
v
[30.3b Independent runner]
|
v
[30.1b Lab data retention]
|
v
[Practice note]
|
v
[Worked example] --> [Exercise EX088]
Learning outcomes
You will (1) run a verification plan against TrackScope; (2) record pass/fail with evidence hashes; (3) stop the lab if a safety test fails.
30.1 Plan
Tests: allow in-scope; deny deny-list; deny bad port; fail-closed missing policy; hours gate; concurrency gate. Each row: id, setup, action, expected, evidence path.
Rows with ids and expected evidence paths make the lab executable by someone who did not write the feature. Fail-closed missing policy sits beside happier allow cases so it cannot be skipped. Sam prints the plan as a checklist the independent runner initials with a UTC timestamp. In one release, a hotfix landed at 14:02 UTC while the lab report showed initials at 14:10 UTC; the timestamp proved the green run was against the hotfix, not the broken build.
30.2 Execution
Use synthetic hosts only. Capture stdout and config hashes. Any safety failure (fail-open) stops the lab and opens a blocker ticket — no "continue to gather data".
"""Evaluate proposed actions against synthetic rules of engagement."""
ROE = {
"hours": ("09:00", "17:00"),
"deny_actions": {"denial-of-service", "phishing-staff", "exfiltrate-real-data"},
"require_approval": {"password-spray-lab", "privilege-escalation-lab"},
"max_concurrency": 5,
}
proposals = [
("service-enum", "10:30", 3, False),
("denial-of-service", "10:30", 1, False),
("password-spray-lab", "10:30", 2, False),
("password-spray-lab", "10:30", 2, True),
("service-enum", "18:00", 2, False),
("service-enum", "11:00", 20, False),
]
def ok_time(t):
return ROE["hours"][0] <= t <= ROE["hours"][1]
for action, when, conc, approved in proposals:
reasons=[]
if action in ROE["deny_actions"]: reasons.append("deny-action")
if action in ROE["require_approval"] and not approved: reasons.append("needs-approval")
if not ok_time(when): reasons.append("outside-hours")
if conc > ROE["max_concurrency"]: reasons.append("concurrency")
print(f"action={action:20} when={when} conc={conc:2} approved={approved} -> {'ALLOW' if not reasons else 'DENY'} {','.join(reasons) or '-'}")
Expected output:
action=service-enum when=10:30 conc= 3 approved=False -> ALLOW -
action=denial-of-service when=10:30 conc= 1 approved=False -> DENY deny-action
action=password-spray-lab when=10:30 conc= 2 approved=False -> DENY needs-approval
action=password-spray-lab when=10:30 conc= 2 approved=True -> ALLOW -
action=service-enum when=18:00 conc= 2 approved=False -> DENY outside-hours
action=service-enum when=11:00 conc=20 approved=False -> DENY concurrency
Synthetic hosts keep the lab inside the course's safety boundary. A fail-open result is a blocker, not a data point to finish later. Capturing config hashes ties the run to a known build. Sam's lab script aborts the remaining rows when any safety row fails, and those blocked rows keep empty evidence paths until a full re-run, so nobody can paste a partial log into a release dossier.
30.3 Report
A one-page lab report: environment version, table of results, open blockers, sign-off. Attach hashes.
Synthetic case — Riverstone Transit. Lab I failed the missing-policy row once; release halted until S-3 was fixed (Chapter 21).
Hashes beat screenshots that cannot be re-checked. The missing-policy failure that halted a release shows why the report's blocker list matters. Sam attaches the report under the same tag as the binary. When Lab I failed the missing-policy row, the returned report went back through the same review queue as a new pack, with the blocker ticket id printed at the top so release managers could not miss why the train halted.
30.4 Errors and limits
Labs on shared clients collide; book the synthetic range.
Shared synthetic ranges need booking the same way production windows do. Collisions produce flaky fails that train people to ignore lab red. Sam's calendar blocks the range and posts the booking link in the lab ticket, and a run without a booking is discarded even if every row looked green. Discarded runs keep logs under a failed tag; last term three failed tags shared the same root cause — two teams booking the synthetic range for the same Friday — which is why the calendar now rejects overlaps.
30.2b Golden fixtures
Keep a sealed directory of synthetic fixtures with known expected decisions. Lab runs must hash-match the fixture set before they count as release evidence. Fixtures drift when someone "fixes" them to match a buggy build — protect them with CODEOWNERS.
CODEOWNERS on fixtures stops quiet edits that hide bugs. Sam's fixture change process requires a paired update to expected decisions and a second reviewer who did not author the build under test, so a buggy binary cannot be hidden by quietly editing the golden answers.
30.3b Independent runner
Lab I is run by someone who did not implement the change under test. Pairing is fine for debugging afterward; the recorded pass needs an independent green checklist.
Independence is about incentives, not distrust of skill. Sam assigns runners from a rotating roster published each Friday for the following week, and the lab lead resolves conflicts so the author of a change is never the recorded runner for that change. The runner's name is recorded in the lab report next to the binary hash.
30.1b Lab data retention
Lab result logs are evidence; retain them with the release tag. Deleting green lab logs to save space destroys the ability to investigate a later regression.
Green logs explain later regressions when a hotfix claims nothing changed. Retention tied to the release tag makes the link mechanical. Sam's delete scripts refuse lab paths still referenced by a living tag, and they print the referencing tag so operators know what would break. A near-miss last quarter paged when an operator tried rm on a lab path still referenced by TrackScope-v0.9.2; the page included the refused command and the user id, and the tag was left intact.
Practice note
Photograph or export the lab environment inventory (package versions, kernel, policy key ids) into the lab report. It passed on my machine without inventory is not transferable evidence. Lab I exists to create transferable confidence for people who were not in the room, including your future self during an incident review six months later.
30.4b What the first lab run proved
The first verification run proved that TrackScope denies every out-of-scope address in the golden fixture set and logs each denial with the rule that fired. It did not prove behaviour under concurrent requests or with malformed scope files; Sam lists those as open items for Laboratory II.
Naming open items, concurrency and malformed files, keeps Laboratory I from overclaiming. Deny-all on golden fixtures is necessary and not sufficient. Sam copies those open items into Laboratory II's plan on day one of the next sprint.
Exercises
- EX088 — Row. Write the fail-closed lab row. Rubric: 3 = complete row; 2 = partial; 1 = vague; 0 = none.
- EX089 — Stop rule. What happens on fail-open? Rubric: 3 = stop, blocker, no engagement use; 2 = note only; 1 = continue; 0 = none.
- EX090 — Evidence. Name three hashes to attach. Rubric: 3 = policy, binary/version, result log; 2 = two; 1 = one; 0 = none.
Synthesis
Verify safety first; stop on fail-open; hash everything.
Continue studying — chapter 30 of 50
← Chapter 29 (draft) · Course chapter index · Chapter 31 (draft) →