◇ OpenCyber Academy
Engineering · Chapter 40 · English

Chapter 40 — Verification laboratory II

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 40 — Verification laboratory II

Lesson map

Lesson 40 published. Map, example, EX118.

flowchart TD
  N0["Learning outcomes"]
  N1["40.1 End-to-end"]
  N2["40.2 Faults"]
  N3["40.3 Sign-off"]
  N4["Practice note"]
  N5["40.1b Observability and parity"]
  N6["Field note — malicious pack"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX118"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["40.1 End-to-end"]
  N2["40.2 Faults"]
  N3["40.3 Sign-off"]
  N4["Practice note"]
  N5["40.1b Observability and parity"]
  N6["Field note — malicious pack"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX118"]

[Learning outcomes]
    |
    v
[40.1 End-to-end]
    |
    v
[40.2 Faults]
    |
    v
[40.3 Sign-off]
    |
    v
[Practice note]
    |
    v
[40.1b Observability and parity]
    |
    v
[Field note — malicious pack]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX118]

Learning outcomes

You will (1) verify end-to-end flow from frame to ticket to coverage; (2) inject faults; (3) sign a release candidate.

40.1 End-to-end

Synthetic frame to monitor decision to event to finding ticket to coverage update. Record hashes at each hop.

In ABP's Lab II, Lena replays a synthetic frame on the bench and follows it through each step: the permit function's decision with its reason code, the alert, the maintenance-system work order, the finding ticket and the coverage update. A hash is recorded at each hop, so the dossier can prove that the ticket describes the same event the monitor saw. A chain that cannot be traced hop by hop is not counted as passing, even when the final result looks right.

40.2 Faults

Kill finding store mid-flow; confirm monitor still denies and drafts queue. Corrupt an evidence file; confirm hash mismatch alert. Feed a flat (unsafe) edge list; confirm reachability fails the SIS check.

# Lab 23 - reachability count on a synthetic Purdue-layered OT network
# Nodes: L5-ERP, L4-Office, L3.5-DMZ, L3-Hist, L2-SCADA, L1-PLC, L1-SIS, L0-IO
edges_flat=[("L5","L4"),("L4","L3"),("L3","L2"),("L2","L1-PLC"),("L2","L1-SIS"),("L1-PLC","L0"),("L1-SIS","L0"),("L4","L2")]
edges_zoned=[("L5","L4"),("L4","L3.5"),("L3.5","L3"),("L3","L2"),("L2","L1-PLC"),("L1-SIS","L0"),("L1-PLC","L0")]  # SIS not reachable from SCADA
def reach(edges, start):
    g={};
    for a,b in edges: g.setdefault(a,[]).append(b)
    seen=set(); stack=[start]
    while stack:
        n=stack.pop()
        if n in seen: continue
        seen.add(n); stack.extend(g.get(n,[]))
    return seen
for name,e in (("flat",edges_flat),("zoned",edges_zoned)):
    r=reach(e,"L4")
    print(f"{name}: L4 reaches {sorted(r)} count={len(r)} writes_to_SIS={'L1-SIS' in r}")

Expected output:

flat: L4 reaches ['L0', 'L1-PLC', 'L1-SIS', 'L2', 'L3', 'L4'] count=6 writes_to_SIS=True
zoned: L4 reaches ['L0', 'L1-PLC', 'L2', 'L3', 'L3.5', 'L4'] count=6 writes_to_SIS=False

Lena's Lab II faults also include collector power loss, which raised the visibility-degraded alert in 40 seconds, and 90 seconds of clock skew on the bench copy of PLC-A. Each fault is injected separately and then in pairs, because a failure that is harmless alone can corrupt evidence when combined with another.

40.3 Sign-off

The monitor release candidate checklist is signed by implementer and reviewer; the process-safety rows must be green.

Synthetic case — Ashford Batch Plant. Lab II found coverage updated before ticket create on retry — fixed ordering with an outbox.

At ABP the Lab II sign-off lists every row with its result and any open defect with an owner, such as the event mis-ordering caused by clock skew. Lena's ordering fix ensures the ticket is written before the coverage board updates, so a retry cannot count a finding that was never recorded.

40.4 Errors and limits

E2E without fault injection only proves the happy path.

Lena's first end-to-end run at ABP passed every row, yet Lab II with faults injected found that clock skew mis-ordered two events in the dossier and that a retry updated coverage before the ticket existed. Neither defect would appear on a clean run, because both depend on timing failures that only occur when something breaks. Lab II sign-off therefore requires each fault script to have run and its result recorded, and a run with no faults is reported as a smoke test.

Practice note

Malicious fixture pack: wrong hashes, reordered events, future timestamps, flat graph claiming to be zoned. Passing only happy fixtures is an incomplete laboratory.

40.1b Observability and parity

Lab II enables debug logs temporarily but seals a redacted copy. Leaving debug on in window builds can leak secrets. Lab environment mirrors production tool versions and policy signing keys (test keys). A Lab II run that passes only on retry blocks the monitor release; it is not rerun until green.

A fault script that cannot run is recorded as a failed row, not skipped. The scripts that inject faults on the bench, such as a dropped policy file, are checked into the lab folder and pinned by SHA in the report. Environment parity gaps explain green-in-lab red-in-window surprises.

Lena's sign-off checks for parity gaps.

Field note — malicious pack

Lab II's malicious pack included a flat graph labelled as zoned. The reachability check failed closed and blocked the release. Happy-path-only laboratories ship lies.

Lena seals the Lab II logs with the same manifest tool as the CNHP dossier, so the bench and the plant use one hashing workflow.

Flaky tests are quarantined and fixed; shipping with sometimes fail-closed is shipping fail-open.

40.5 Worked example — end-to-end fault injection (synthetic ABP)

Laboratory II exercises the full chain on the bench: replayed traffic → permit function → alert → maintenance-system work order → dossier. Injected faults: collector power loss, clock skew of 90 seconds on PLC-A, a malformed event, and a dead-letter backlog. Results: power loss raised "visibility degraded" in 40 s; clock skew was detected by the time-source check but mis-ordered two events in the dossier until the offset correction was applied; the malformed event reached the dead-letter queue; the backlog cleared in 7 minutes. Sign-off lists the mis-ordering as an open defect with an owner.

Exercises

Synthesis

E2E with faults; green safety; two signatures.

Continue studying — chapter 40 of 50

← Chapter 39 (draft) · Course chapter index · Chapter 41 (draft) →