Chapter 30 — Verification laboratory I
Lesson map
Lesson 30 published. Map, example, EX088.
flowchart TD
N0["Learning outcomes"]
N1["30.1 Verification versus validation"]
N2["30.2 The laboratory"]
N3["Lab — unit tests for the burst rule"]
N4["30.3 Reading the result"]
N5["30.4 Pipeline tests"]
N6["30.5 Evidence"]
N7["30.6 Repeatability"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX088"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["30.1 Verification versus validation"]
N2["30.2 The laboratory"]
N3["Lab — unit tests for the burst rule"]
N4["30.3 Reading the result"]
N5["30.4 Pipeline tests"]
N6["30.5 Evidence"]
N7["30.6 Repeatability"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX088"]
[Learning outcomes]
|
v
[30.1 Verification versus validation]
|
v
[30.2 The laboratory]
|
v
[Lab — unit tests for the burst rule]
|
v
[30.3 Reading the result]
|
v
[30.4 Pipeline tests]
|
v
[30.5 Evidence]
|
v
[30.6 Repeatability]
|
v
[Worked example] --> [Exercise EX088]
Learning outcomes
You will (1) verify detection logic with automated unit tests; (2) verify pipeline behaviour with safe synthetic events; (3) record verification evidence; (4) distinguish verification from validation.
30.1 Verification versus validation
Verification asks "did we build it right?" — does the rule behave as specified? Validation asks "did we build the right thing?" — does it help detect real threats in operation? This laboratory focuses on verification.
A rule can pass every verification test and still be useless if it targets behaviour attackers do not use, or if real attacks look different from the test data. Validation answers that question later, with operational evidence. Keeping the two separate stops a team from claiming a rule 'works' when they have only shown it matches its specification.
30.2 The laboratory
The laboratory is a test SIEM isolated from production, with replayed synthetic logs. No real customer data and no attack tools are used; test events are crafted log records that look like the behaviour the rule targets.
Isolation stops test events from reaching production analysts and wasting their time. Replayed synthetic logs give realistic volume and variety without exposing customer data. Crafted records let the tester control exactly which conditions are present, which is how boundary cases are tested precisely. The laboratory is rebuilt from configuration so its state is known before each run.
Lab — unit tests for the burst rule
"""Unit tests for a detection function, run with the standard library (synthetic cases)."""
import io, unittest
def burst(users_failed, any_success, threshold=5):
return users_failed >= threshold and any_success
class T(unittest.TestCase):
def test_fires_at_threshold(self): self.assertTrue(burst(5, True))
def test_quiet_below(self): self.assertFalse(burst(4, True))
def test_needs_success(self): self.assertFalse(burst(9, False))
def test_zero(self): self.assertFalse(burst(0, False))
if __name__ == "__main__":
r = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(unittest.defaultTestLoader.loadTestsFromTestCase(T))
print("tests run", r.testsRun, "failures", len(r.failures))
Run with python3 lab_unittest.py. Verified output:
tests run 4 failures 0
30.3 Reading the result
Four tests run and none fail: the rule fires at exactly 5 distinct failing users with a success, stays quiet at 4, stays quiet without a success even at 9 users, and stays quiet with no activity. These encode the boundary and the success clause decided in chapters 9 and 24.
Each test checks one aspect of behaviour. Firing at five and staying quiet at four pins down the threshold exactly; staying quiet at nine without a success shows that the success clause really is required; staying quiet with no activity shows the rule does not misfire on empty windows. Four passing tests together describe the rule's intended behaviour.
30.4 Pipeline tests
Unit tests check logic; pipeline tests check that a synthetic record entering the collector produces an alert in the case system with correct fields. Amara injects three records and checks each field in the resulting alert.
A rule can be correct while the pipeline around it breaks: a field renamed by the parser, a routing rule sending alerts to the wrong queue, or a time conversion error. Checking every field in the final alert, such as customer ID, source network and subsequent actions, confirms the whole chain works, not just the rule's logic.
30.5 Evidence
The laboratory record lists the test, version, input, expected and actual results, date and tester.
Recording the rule version ties each result to an exact configuration, so a later change can be compared. Expected and actual results side by side make failures obvious. The tester's name and date show accountability. These records become part of the evidence that the rule was verified before going live. Failed tests stay in the record after they are fixed, alongside the rerun, so the history shows what went wrong and how it was corrected.
30.6 Repeatability
Tests run automatically on each rule change, so verification is continuous, not a one-off.
Rules change often through tuning, parser updates and field renames. A test run only once at creation says nothing about the rule six months later. Running the suite on every change catches regressions immediately, and a failing test blocks deployment until someone fixes the rule or justifies changing the test. The suite takes under two minutes, so engineers run it locally before proposing a change, and the shared pipeline runs it again before merging.
Test data hygiene
Synthetic test records carry a marker field so they never mix with real alerts or metrics.
Worked example
The pipeline test reveals that the alert lacks the customer ID: the routing template referenced an old field name. Amara fixes the template, re-runs, and records both the failure and the fix. The unit tests alone would never have found this.
Synthetic case. Tests and records are synthetic.
Errors and limits
Passing verification does not show the rule catches real attackers; that is validation, covered in chapter 38.
Exercises
- EX088 — V and V. Distinguish verification from validation. Rubric: 3 = built right vs right thing; spec vs real effectiveness; 2 = partial; 1 = same; 0 = none.
- EX089 — Four tests. What do the four unit tests check? Rubric: 3 = threshold boundary 5/4, success clause, empty input; 2 = three; 1 = one; 0 = none.
- EX090 — Pipeline finding. What did the pipeline test catch that unit tests could not? Rubric: 3 = missing customer ID from template field name; 2 = partial; 1 = vague; 0 = none.
Synthesis
Verify logic with unit tests and the whole pipeline with safe synthetic events, recording every result.
Continue studying — chapter 30 of 50
← Chapter 29 (draft) · Course chapter index · Chapter 31 (draft) →