◇ OpenCyber Academy
Engineering · Chapter 30 · English

Chapter 30 — Verification laboratory I

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 30 — Verification laboratory I

Lesson map

Lesson 30 published. Map, example, EX088.

flowchart TD
  N0["Learning outcomes"]
  N1["30.1 Verification versus validation"]
  N2["30.2 Procedures"]
  N3["30.3 Coverage"]
  N4["30.4 Deviations"]
  N5["30.5 Evidence"]
  N6["Regression suite"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX088"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["30.1 Verification versus validation"]
  N2["30.2 Procedures"]
  N3["30.3 Coverage"]
  N4["30.4 Deviations"]
  N5["30.5 Evidence"]
  N6["Regression suite"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX088"]

[Learning outcomes]
    |
    v
[30.1 Verification versus validation]
    |
    v
[30.2 Procedures]
    |
    v
[30.3 Coverage]
    |
    v
[30.4 Deviations]
    |
    v
[30.5 Evidence]
    |
    v
[Regression suite]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX088]

Learning outcomes

You will (1) write verification procedures that trace to requirements; (2) execute them on synthetic data; (3) record pass, fail and deviation results; (4) distinguish verification ("built right") from validation ("right thing").

30.1 Verification versus validation

Verification checks that Lantern meets its written requirements. Validation, in Chapter 38, checks that it meets stakeholders' real needs. A platform can pass verification and still fail validation if the requirements were wrong.

Kofi keeps the two activities in separate plans with separate owners. Verification is run by the engineering team against the requirements document. Validation involves the people who use Lantern and asks whether it helps their work. A requirement that passes verification but fails validation is evidence that the requirement itself needs changing.

30.2 Procedures

Each procedure names the requirement, preconditions, steps, expected result and pass criteria. "Load bundle S-07 containing 1,000 indicators of which 12 lack valid_from; expect 988 stored, 12 rejected with reason; pass if counts match exactly."

The example shows the level of detail needed: a named bundle, exact counts and a clear pass rule. Two people following it should get the same result. Kofi writes the procedure before running it, so the expected result cannot be shaped by what the system happens to produce on the day.

30.3 Coverage

The traceability matrix shows which requirements have procedures. Uncovered requirements are listed explicitly; an untested requirement is an open risk, not a silent pass.

Kofi's coverage report lists every requirement with its procedures and last result. At the first run, several requirements had no procedure, mostly non-functional ones such as retention. Each gap was either filled with a procedure or recorded as an accepted risk with an owner and a date for review. Nobody was allowed to mark an untested requirement as met.

30.4 Deviations

When a result differs, record a deviation with severity and owner. Do not edit the expected result to match the output without a requirement change.

Each deviation records the procedure, what was expected, what happened and the evidence. Severity follows the effect on users: a TLP gate failure is high because it could leak restricted data; a missing metric is lower. Kofi reviews open deviations weekly with the team, and closes each only after the procedure has been rerun and passed.

30.5 Evidence

Logs, metric snapshots and output hashes are stored with the procedure run. A reviewer can rerun the same procedure with the same synthetic data and compare hashes.

Hashes make comparison quick: if the output file has the same hash as the stored run, the result is identical. Kofi keeps the synthetic data, the code version and the configuration with each run, so a reviewer months later can reproduce it exactly. Evidence that cannot be reproduced has much less weight in an audit.

Regression suite

Passed procedures become automated regression tests run on every change. The suite grows with each defect fixed, so the same mistake cannot quietly return. Kofi tags each test with its requirement ID, which keeps the traceability matrix current without manual effort and shows at a glance which requirements lost coverage when a test was removed.

Worked example

Kofi runs 42 procedures. 38 pass, 3 fail (TLP gate allowed a label with trailing space; reject reason missing for one error type; checkpoint metric not exported), and 1 is blocked by a missing test fixture. Each failure becomes a defect with an owner; the blocked one is scheduled.

Synthetic case. Counts and defects are invented.

Errors and limits

A green test run on synthetic data does not prove robustness on every partner format; extend fixtures as formats are seen. Do not count blocked procedures as passes.

Exercises

Synthesis

Verify against written requirements with traceable procedures, record deviations honestly and keep evidence rerunnable.

Continue studying — chapter 30 of 50

← Chapter 29 (draft) · Course chapter index · Chapter 31 (draft) →