Chapter 40 — Verification laboratory II
Lesson map
Lesson 40 published. Map, example, EX118.
flowchart TD
N0["Learning outcomes"]
N1["40.1 Non-functional verification"]
N2["40.2 Fault injection"]
N3["40.3 Negative tests for controls"]
N4["40.4 Verification report"]
N5["40.5 Re-verification"]
N6["Practice note — evidence packaging"]
N7["Residual risk"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX118"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["40.1 Non-functional verification"]
N2["40.2 Fault injection"]
N3["40.3 Negative tests for controls"]
N4["40.4 Verification report"]
N5["40.5 Re-verification"]
N6["Practice note — evidence packaging"]
N7["Residual risk"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX118"]
[Learning outcomes]
|
v
[40.1 Non-functional verification]
|
v
[40.2 Fault injection]
|
v
[40.3 Negative tests for controls]
|
v
[40.4 Verification report]
|
v
[40.5 Re-verification]
|
v
[Practice note — evidence packaging]
|
v
[Residual risk]
|
v
[Worked example] --> [Exercise EX118]
Learning outcomes
You will (1) verify non-functional requirements — performance, availability, security; (2) run fault-injection tests safely; (3) verify TLP and privacy controls with negative tests; (4) compile a verification report.
40.1 Non-functional verification
Performance: inject synthetic bundles at 150% of peak rate and verify time-to-enforcement percentiles. Availability: kill one enrichment instance and verify continued service. Security: attempt restricted exports with test accounts lacking permission.
Kofi chose the load figure to cover a busy campaign day with margin. The availability test stops one instance while synthetic traffic continues, and checks that the time-to-enforcement percentiles stay within target. The security tests use accounts created for testing, so their permissions are known exactly, and every attempt is logged.
40.2 Fault injection
Faults are injected in staging: partner stub returns errors, clock skew is introduced, storage fills up. Each test states the expected degraded behaviour and verifies it. Fault injection never targets production or partner systems.
Kofi's fault list came from the failure analysis: each high-ranked failure mode has at least one injection test. The clock skew test in the worked example found a real problem that would otherwise have shown up only in production. Tests restore the normal state afterwards and check that recovery is complete.
40.3 Negative tests for controls
Try to export a RED report to a non-named recipient; try to push an OT rule without approval; try to store a personal social-media profile via the OSINT module. Every attempt must fail and be logged.
Kofi writes the expected log entry for each test before running it, so a test that fails but leaves no log counts as a failure too. The OSINT test uses an invented profile on the local stub, so no real platform is contacted. Logs are checked afterwards.
40.4 Verification report
Summary of procedures executed, passes, failures, deviations, blocked items, open defects with severity and owners, and residual risks. The report goes to the design review in Chapter 45.
Kofi writes the summary for readers who will not open the procedures. It starts with the overall result and the residual risks, then lists failures with their severity. Blocked items are reported separately from passes, because a procedure that could not run tells the reviewers nothing about the requirement. The full procedures and evidence are linked for anyone who wants to check a claim.
40.5 Re-verification
After fixes, failed procedures rerun, and the regression suite runs in full.
Running the full suite, not just the failed procedures, catches fixes that break something else. After the clock fix in the worked example, Kofi reran the skew test and then the whole regression suite. Both passed, and the report records the rerun with its date, the code version and the evidence hashes. A fix that has not been re-verified stays open in the defect list, however confident the engineer who made it may be.
Practice note — evidence packaging
Each procedure's logs and outputs are hashed and stored with the report, so reviewers can check that the evidence matches what the report claims.
Residual risk
The report closes with risks that verification could not address, such as behaviour on partner formats never seen in testing, so reviewers weigh them explicitly rather than assuming coverage. Each residual risk names an owner and a monitoring approach.
Worked example
At 150% load, p95 time-to-enforcement is 22 minutes against a 30-minute requirement: pass. The clock-skew test shows sightings timestamps off by five minutes: fail; NTP monitoring is added. All six negative control tests block and log correctly.
Synthetic case. Results are invented.
Errors and limits
Staging differs from production; record differences. Passing negative tests shows these attacks fail, not that no attack succeeds.
Exercises
- EX118 — Negative test. Write one negative test for the OSINT privacy requirement. Rubric: 3 = attempt to store personal profile → expect rejection + log; 2 = test without expected result; 1 = positive test; 0 = none.
- EX119 — Load verdict. p95 is 22 min against a 30-min requirement. Verdict and reason? Rubric: 3 = pass; 22 ≤ 30 at stated percentile; 2 = pass; 1 = fail; 0 = none.
- EX120 — Injection target. Where may fault injection run, and where must it never run? Rubric: 3 = staging/synthetic stubs; never production or partner systems; 2 = one half; 1 = production; 0 = none.
Synthesis
Verify performance, availability and controls with load, faults and negative tests in staging, and package evidence.
Continue studying — chapter 40 of 50
← Chapter 39 (draft) · Course chapter index · Chapter 41 (draft) →