Chapter 40 — Verification laboratory II | Engineering Authorized offensive security and red team

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 40 — Verification laboratory II

Lesson map

Lesson 40 published. Map, example, EX118.

flowchart TD
  N0["Learning outcomes"]
  N1["40.1 End-to-end"]
  N2["40.2 Faults"]
  N3["40.3 Sign-off"]
  N4["40.2b Observability in the lab"]
  N5["40.3b Flaky tests"]
  N6["40.1b Environment parity"]
  N7["Practice note"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX118"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["40.1 End-to-end"]
  N2["40.2 Faults"]
  N3["40.3 Sign-off"]
  N4["40.2b Observability in the lab"]
  N5["40.3b Flaky tests"]
  N6["40.1b Environment parity"]
  N7["Practice note"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX118"]

[Learning outcomes]
    |
    v
[40.1 End-to-end]
    |
    v
[40.2 Faults]
    |
    v
[40.3 Sign-off]
    |
    v
[40.2b Observability in the lab]
    |
    v
[40.3b Flaky tests]
    |
    v
[40.1b Environment parity]
    |
    v
[Practice note]
    |
    v
[Worked example] --> [Exercise EX118]

Learning outcomes

You will (1) verify end-to-end flow from action to ticket to coverage; (2) inject faults; (3) sign a release candidate.

40.1 End-to-end

Synthetic action → TrackScope decision → event → FindingsDesk ticket → PulseMap coverage update. Record hashes at each hop.

Unit greens can still hide a broken event path into FindingsDesk. End-to-end on synthetic fixtures exercises allow, deny, fail-closed and a full finding upsert before a release candidate is cut. Sam's Lab II script is the gate named in Capstone II. Capstone II cannot cite a unit-only green as Lab II evidence; the script must print the FindingsDesk finding id so the checklist can link it.

40.2 Faults

Kill FindingsDesk mid-flow; confirm TrackScope still denies and drafts queue. Corrupt an evidence file; confirm hash mismatch alert.

"""Verify integrity of a synthetic evidence package."""
import hashlib, json
files = {
    "roe.pdf": b"SYNTHETIC-ROE-v1",
    "scope.csv": b"host,port\n10.20.1.10,443\n",
    "findings.json": b'[{"id":"SYN-001"}]',
}
expected = {n: hashlib.sha256(b).hexdigest()[:16] for n,b in files.items()}
# Simulate tampering of findings.json
current = dict(expected)
current["findings.json"] = hashlib.sha256(b'[{"id":"SYN-001"},{"id":"TAMPERED"}]').hexdigest()[:16]
print(json.dumps(expected, indent=1))
for n in sorted(expected):
    status = "OK" if expected[n] == current[n] else "MISMATCH"
    print(f"{n:16} {status}")

Expected output:

{
 "roe.pdf": "18d74983d403e7db",
 "scope.csv": "01cc4815eddf3de6",
 "findings.json": "7f5f178ab43ef00f"
}
findings.json    MISMATCH
roe.pdf          OK
scope.csv        OK

Injecting a missing policy file and a dead FindingsDesk in the lab is how safe degradation stays proven. The expected outcome is deny-all and queued drafts, never an allow. Sam fails Lab II if either inject produces an allow or a lost draft count. The fourteen-draft outage story is the fixture size used for the dead-FindingsDesk inject. Operators watch queue depth climb to fourteen and fall to zero during replay as part of the scored run.

40.3 Sign-off

Release candidate checklist signed by implementer and reviewer; safety rows must be green.

Synthetic case — Riverstone Transit. Lab II found coverage updated before ticket create on retry — fixed ordering with outbox.

Independent sign-off is the same incentive rule as Lab I: the implementer does not green their own release. The checklist names tool SHAs, fixture hashes and the fault-inject results. Sam's release notes link the signed checklist, not a screenshot of a local demo. An implementer who signs their own checklist is rejected by the release script before the tag is created.

40.4 Errors and limits

E2E without fault injection only proves the happy path.

Flaky end-to-end runs train people to re-click until green. Quarantine flakes, fix them, or delete the test — do not skip Lab II. Sam's last flake was a clock skew in the hours gate; it blocked two releases until the fixture used fixed UTC times. A flake on a fail-closed row now opens a P1 with the same severity as a production deny bug. Re-clicking until green is recorded as a process incident when it touches those rows.

40.2b Observability in the lab

Lab II enables debug logs temporarily but seals a redacted copy. Leaving debug on in engagement builds can leak tokens into log aggregates. Toggle is part of the sign-off checklist.

If the lab cannot show deny reason codes and queue depth, operators will be blind in a real outage. Lab II asserts those dashboards update during the fault injects. Sam watches the same panels in the synthetic range that on-call will watch on a Friday night. If the reason-code panel stays empty during a deny inject, Lab II fails even when the HTTP status looks right.

40.3b Flaky tests

A flaky Lab II is a blocker, not a retry-until-green ritual. Quarantine the test, fix the race (clocks, ordering), then re-run. Shipping with "sometimes fail-closed" is shipping fail-open.

A flake that touches fail-closed rows is a P1, not a statistical curiosity. Quarantine removes it from the merge gate only with an owner and a due date. Sam's board shows quarantined safety tests in red until they return stable for a week of runs. Quarantine without a due date is rejected by the board automation.

40.1b Environment parity

Lab II environment mirrors production tool versions and policy signing keys (test keys). Parity gaps explain "green in lab, red in engagement" surprises.

Lab II on a laptop layout that differs from the jump host proves the laptop, not the engagement path. Parity means the same policy loader, bucket config and event sink type. Sam's lab image is rebuilt from the same golden branch as the jump host each Monday. A laptop-only pass is labelled demo and cannot satisfy Capstone II.

Practice note

Lab II includes a malicious fixture pack: wrong hashes, reordered events, future timestamps. Passing only happy fixtures is an incomplete laboratory. The malicious pack is versioned beside the golden fixtures and covered by CODEOWNERS so it cannot be weakened quietly to make a release green.

Seal Lab II logs with the same manifest tool the dossier uses so students practise one hashing workflow.

Fault-injection scripts are checked into the lab folder and pinned by SHA in the report.

Happy-path green with malicious-pack red still fails Lab II sign-off.

40.4b Sign-off evidence

The Laboratory II sign-off pack holds the test plan, the run logs, the list of injected faults with observed behaviour, and the open issues with owners. An independent engineer signs only after reproducing two randomly chosen results from the pack.

The signed checklist, fixture hashes and fault-inject logs are sealed next to the binary tag. Without them, "Lab II green" is a rumour. Sam's artefact store rejects a release tag that lacks the checklist hash in its metadata. Missing metadata is how a hallway green once almost shipped; the store check stopped it.

Exercises

EX118 — Ordering. Why create ticket before coverage count increases? Rubric: 3 = coverage must reflect recorded findings; 2 = vague; 1 = either order; 0 = none.
EX119 — Fault. Expected TrackScope behaviour when desk is down. Rubric: 3 = still evaluate/deny; queue drafts; 2 = partial; 1 = fail open; 0 = none.
EX120 — Sign-off. Two signatures required — who? Rubric: 3 = implementer + independent reviewer; 2 = one; 1 = manager only; 0 = none.

Synthesis

E2E with faults; green safety; two signatures.

Continue studying — chapter 40 of 50

← Chapter 39 (draft) · Course chapter index · Chapter 41 (draft) →