Chapter 09 — Programming II
Lesson map
Lesson 09 published. Map, example, EX018.
flowchart TD
N0["Learning outcomes and connection"]
N1["9.1 Structured data and modules"]
N2["9.2 Exceptions as data"]
N3["9.3 Provenance"]
N4["9.4 Property-based testing"]
N5["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX018"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["9.1 Structured data and modules"]
N2["9.2 Exceptions as data"]
N3["9.3 Provenance"]
N4["9.4 Property-based testing"]
N5["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX018"]
[Learning outcomes and connection]
|
v
[9.1 Structured data and modules]
|
v
[9.2 Exceptions as data]
|
v
[9.3 Provenance]
|
v
[9.4 Property-based testing]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX018]
Learning outcomes and connection
After this chapter you can (1) structure an event-normalisation component as modules with clear interfaces; (2) handle errors as data rather than control-flow accidents; (3) write property-based tests that explore malformed and duplicate input; and (4) carry provenance through every transformation. It builds on Chapter 4 and prepares Chapters 14, 22 and 30.
9.1 Structured data and modules
A normaliser converts heterogeneous events into a common schema. Design it as a pipeline of small pure functions — parse, validate, map fields, enrich, emit — each with a typed input and output. Modules should hide decisions likely to change behind stable interfaces (Parnas, 1972); the mapping from a vendor's field names to your schema is such a decision. In the HVSS normaliser, the identity-provider field mapping lives in its own module with a version number, so a vendor rename changes one file and one mapping version.
9.2 Exceptions as data
Inside one function, an exception is fine. At the pipeline boundary, every failure becomes a record: {"input_ref": ..., "stage": "parse", "error": "json_decode", "detail": ...} routed to a dead-letter store. The pipeline never loses an input by raising past its caller. Retries apply only to transient errors (a dependency timeout), never to deterministic ones (malformed JSON), or you build an infinite loop.
In the HVSS burst, the truncated identity-provider events are deterministic failures: retrying a json_decode error on the same bytes will fail every time. A timeout from the user directory lookup in EX018 is transient, so it can be retried with a limit before the event goes on with its enrichment marked missing.
9.3 Provenance
Every output event carries the hash of its raw input, the normaliser version and the mapping version. Then an analyst who sees an odd field can ask "which rule produced this?" and get an answer. Provenance is cheap at write time and impossible to reconstruct later. The W3C PROV model (PROV-Overview) gives vocabulary — entity, activity, agent — that maps directly: raw event (entity), normalisation (activity), normaliser version (agent). For the HVSS normaliser, those three fields would have shown at once which version had silently skipped the truncated identity-provider events.
9.4 Property-based testing
Example tests check cases you thought of; property tests check statements over many generated inputs. Useful properties for a normaliser: conservation (outputs + dead letters = inputs); idempotence (normalising an already-normalised event is the identity); determinism (same input, same output); provenance (every output's raw hash matches a stored raw input). A minimal stdlib generator:
import random, json
def gen(rng):
base = {"ts": "2026-03-02T08:00:00Z", "user": "u1", "action": "login"}
m = rng.choice(["ok", "drop_key", "bad_json", "dup", "unicode", "huge"])
if m == "drop_key": base.pop(rng.choice(list(base)))
if m == "unicode": base["user"] = "ü\u202e1" # includes a right-to-left override
if m == "huge": base["action"] = "x" * 100_000
s = json.dumps(base)
return s[:-3] if m == "bad_json" else s
Run it with fixed seeds, assert the properties and keep failing seeds as regression tests. The right-to-left override character is a reminder that display-confusing input is an attack technique, not just a curiosity.
9.5 Worked example (synthetic)
HVSS's normaliser receives a burst in which 3% of identity-provider events are truncated JSON and 1% are exact duplicates. Version 1 caught the decode error with a bare except and continued — conservation failed by 3%, unnoticed for weeks. Version 2 routed errors to a dead-letter store with counters exported as metrics; the 3% became an alert and a vendor ticket on the first day.
Version 1 failed the conservation property of the previous subsection, because truncated inputs produced neither an output nor a dead letter. A property test with the bad_json generator mode would have exposed the gap before deployment.
9.6 Errors and limits
- Bare exception handlers that continue silently.
- Retrying deterministic failures.
- Dropping unknown fields (Chapter 24 shows the consequence).
- Property tests with random seeds that are not recorded.
Each error in this list maps onto a step in the HVSS normaliser history. The bare except in version 1 is the first error, and the weeks of unnoticed 3% loss are its cost. Retrying the truncated JSON would add load without ever succeeding. A failing generator seed that was not recorded cannot be replayed as a regression test after the fix, so the property test proves less than it seems.
Exercises — chapter 09
EX017 — Properties. Objective: specify testable properties. Inputs: the generator above. Steps: write four properties as assertions for a normaliser norm(s) -> (event | None, error | None); state which generator mode exercises each. Expected evidence: assertions and mapping. Rubric: 3 = conservation, determinism, idempotence and provenance or display-safety correctly mapped; 2 = three; 1 = vague; 0 = none.
EX018 — Dead letters. Objective: design failure handling. Inputs: normaliser with a dependency (a user directory lookup) that times out 2% of the time. Steps: specify retry policy, dead-letter schema and the metric that would alert operators. Expected evidence: specification. Rubric: 3 = bounded retries with backoff for timeouts only, enrichment marked missing rather than event dropped, metric with threshold; 2 = partial; 1 = generic; 0 = none.
Chapter synthesis
Robust response software treats failure as data, carries provenance on every record and is tested with properties over generated hostile input. These habits are what make later pipeline guarantees provable rather than hoped for.
Continue studying — chapter 09 of 50
← Chapter 08 · Course chapter index · Chapter 10 →
Continue to the next authored lesson.