Chapter 30 — Verification laboratory I

Lesson map

Lesson 30 published. Map, example, EX088.

flowchart TD
  N0["Learning outcomes"]
  N1["30.1 Test plan"]
  N2["30.2 Running tests"]
  N3["30.3 A failed test"]
  N4["30.4 Test versus control"]
  N0 --> N1 --> N2 --> N3 --> N4
  N4 --> EX["Worked example"]
  EX --> Q["Exercise EX088"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["30.1 Test plan"]
  N2["30.2 Running tests"]
  N3["30.3 A failed test"]
  N4["30.4 Test versus control"]
  N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX088"]

[Learning outcomes]
    |
    v
[30.1 Test plan]
    |
    v
[30.2 Running tests]
    |
    v
[30.3 A failed test]
    |
    v
[30.4 Test versus control]
    |
    v
[Worked example] --> [Exercise EX088]

Learning outcomes

You will (1) verify controls with planned tests; (2) record test evidence; (3) distinguish test failure from control failure.

30.1 Test plan

In the verification laboratory, Lindiwe writes a test plan for four controls she helped build: the baseline check, the password-guessing measure, the address-change alert and the asset reconciliation. For each, she defines inputs, expected results and pass criteria. The baseline check should fail a planted server with root login enabled; the guessing measure should flag a planted burst of ten failures in four minutes. Prof. Hale approves the plan before testing starts. Each test is listed with its planted input and the expected outcome in one table.

30.2 Running tests

Lindiwe runs the tests on synthetic data. The baseline check correctly fails the planted server. The guessing measure flags the planted burst and ignores a planted slow sequence, as intended. The address alert fires for a planted change one hour before dispatch. The asset reconciliation correctly lists a planted unregistered host. She records each run with date, inputs, outputs and the version of each script, so results can be reproduced. All four controls pass their first planned cases. She stores the planted inputs alongside the scripts.

30.3 A failed test

One test fails: a planted burst of exactly ten failures spanning precisely five minutes is not flagged. Lindiwe examines the code and finds the comparison treats the boundary inconsistently with her written definition. She corrects the code, reruns all tests and confirms the planted case is now flagged and the earlier results unchanged. Prof. Hale notes that boundary cases are where definitions and code most often disagree. She adds the boundary case to the permanent test set, so the same mistake would be caught again. The fix is a single changed comparison.

30.4 Test versus control

Lindiwe reflects on what the failed test meant. The control itself, detecting guessing, was sound; the implementation had a boundary error. Had the test not existed, a real attack exactly at the boundary would have been missed. She writes in her laboratory report that tests verify implementations against definitions, and that keeping planted test cases permanently lets future changes be checked quickly. Prof. Hale asks her to present this example to the class, because it shows clearly why verification needs planned cases at every boundary rather than only typical inputs.

Worked example

Lindiwe tests four controls with planted cases, finds a boundary error in the guessing measure, fixes it and reruns all tests.

Synthetic case. Tests are synthetic.

Errors and limits

Untested boundary cases often reveal disagreement between definition and code.

Exercises

• EX088 — Planted burst. What planted burst should be flagged?
  Show marking criteria

  Rubric: 3 = ten failures in four minutes;; 2 = partly correct;; 1 = vague;; 0 = none.
• EX089 — Failing test. Which planted case failed at first?
  Show marking criteria

  Rubric: 3 = ten failures spanning exactly five minutes;; 2 = partly correct;; 1 = vague;; 0 = none.
• EX090 — Permanent tests. Why keep planted cases permanently?
  Show marking criteria

  Rubric: 3 = to check future changes quickly;; 2 = partly correct;; 1 = vague;; 0 = none.