Chapter 40 — Verification laboratory II
Lesson map
Lesson 40 published. Map, example, EX118.
flowchart TD
N0["Learning outcomes"]
N1["40.1 Full regression"]
N2["40.2 Incident scenario"]
N3["40.3 Failure tests"]
N4["40.4 Evidence"]
N5["Regression after every change"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX118"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["40.1 Full regression"]
N2["40.2 Incident scenario"]
N3["40.3 Failure tests"]
N4["40.4 Evidence"]
N5["Regression after every change"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX118"]
[Learning outcomes]
|
v
[40.1 Full regression]
|
v
[40.2 Incident scenario]
|
v
[40.3 Failure tests]
|
v
[40.4 Evidence]
|
v
[Regression after every change]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX118]
Learning outcomes
You will (1) run a full regression and failure suite; (2) verify against the incident scenario; (3) record evidence.
40.1 Full regression
At the end of implementation, Kofi runs the complete regression suite in the lab against final configurations generated from the templates for every office. The suite contains positive tests showing permitted flows work, negative tests showing forbidden flows fail, and boundary tests on ports and addresses next to permitted ones. Running it against every office, not just a sample, confirms that per-site values have not introduced site-specific errors. All tests pass, and the results are recorded with the configuration versions used, so the evidence can be reproduced later.
40.2 Incident scenario
He then reconstructs the incident that started the project, entirely in the lab with synthetic devices: a test machine on the guest network attempts the same kinds of file-sharing connections towards internal zones that the infected laptop made. Every attempt is denied and logged, and because the number of attempts exceeds the deny-spike threshold, an alert reaches the test on-call address within two minutes. In the old flat network, the same behaviour went unnoticed for hours. The comparison is the clearest single piece of evidence that the project addressed its original problem.
40.3 Failure tests
Security that fails during an outage is not enough, so the lab also tests failures. Kofi disables one tunnel at a branch and confirms traffic moves to the second within the requirement. He powers off the active main firewall and confirms the standby takes over with existing connections preserved. He issues a test certificate with a short life and confirms that monitoring flags it sixty days before expiry. Each test has a defined expected result written before it runs, so the outcome is judged against a prediction rather than interpreted afterwards.
40.4 Evidence
Each test result is recorded with the date, the tester, the configuration and template versions, the expected result and the actual result. Failed results are kept alongside the later passing ones, with a note of the fix. All records go into the evidence dossier, indexed by requirement. Recording evidence this carefully may seem excessive for a lab, but it allows an examiner or auditor to see exactly what was tested against which configuration, and it allows the council to rerun the same tests after future changes and compare results directly.
Regression after every change
The regression suite is not run only once at the end. Kofi wires it into the change process so that every policy change is tested in the lab against all positive, negative and boundary tests before deployment. This matters because a rule added to fix one flow can unintentionally open another, especially when rule order decides the outcome. Running the whole suite takes about twelve minutes, which is short enough that engineers do not skip it. Over the project, the suite catches three changes that would have broken an existing requirement, each fixed before reaching any office. The suite itself is versioned with the configurations.
Limits of laboratory evidence
Lab results are strong evidence but not complete proof. The lab uses the same firewall models and generated configurations, yet real offices have traffic patterns, cabling faults and user behaviour the lab cannot reproduce. Kofi therefore pairs each lab result with a production observation where possible: the failover test is repeated in a maintenance window, deny patterns are compared with lab predictions, and latency is measured on live links. Where production confirmation is impossible or unsafe, as with the incident reconstruction, he states clearly that the evidence is lab-only. Being explicit about this difference is part of honest engineering reporting and helps examiners judge the strength of each claim.
Worked example
The suite runs 96 tests with all passing; the incident scenario produces an alert in 1 minute 40 seconds.
Synthetic case. Lab results are synthetic.
Errors and limits
Reconstructing incidents must be done only in the lab with synthetic devices.
Exercises
- EX118 — Incident result. What happened in the reconstructed incident? Rubric: 3 = denied, logged, alerted under two minutes;; 2 = blocked;; 1 = vague;; 0 = none.
- EX119 — Failure tests. Name three failure tests. Rubric: 3 = tunnel, firewall, certificate;; 2 = two;; 1 = one;; 0 = none.
- EX120 — Lab-only reconstruction. Why reconstruct the incident only in the lab? Rubric: 3 = authorisation and safety;; 2 = safe;; 1 = vague;; 0 = none.
Synthesis
The final lab suite shows the design blocks the original incident and survives failures.
Continue studying — chapter 40 of 50
← Chapter 39 (draft) · Course chapter index · Chapter 41 (draft) →