Chapter 14 — Software engineering
Lesson map
Lesson 14 published. Map, example, EX027.
flowchart TD
N0["Learning outcomes and connection"]
N1["14.1 Interfaces and versions"]
N2["14.2 Tests that protect evidence"]
N3["14.3 Review, deployment and rollback"]
N4["14.5 Release plan"]
N5["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX027"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["14.1 Interfaces and versions"]
N2["14.2 Tests that protect evidence"]
N3["14.3 Review, deployment and rollback"]
N4["14.5 Release plan"]
N5["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX027"]
[Learning outcomes and connection]
|
v
[14.1 Interfaces and versions]
|
v
[14.2 Tests that protect evidence]
|
v
[14.3 Review, deployment and rollback]
|
v
[14.5 Release plan]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX027]
Learning outcomes and connection
After this chapter you can (1) define interfaces and versioning for response tooling; (2) write acceptance and regression tests that detect evidence loss; (3) design review, deployment and rollback so a change cannot silently erase evidence or bypass controls; and (4) analyse a regression to its root cause. It builds on Chapters 4 and 9 and prepares Chapters 24, 37 and 39.
14.1 Interfaces and versions
An evidence-ingest service has consumers — analysts, detection rules, dashboards, legal hold — who depend on its output schema. Treat the schema as a published interface with semantic versions: a removed or renamed field is a breaking change requiring a major version and a migration plan. Consumers should declare which version they expect, so a mismatch fails loudly. Brooks (1987) argued that the essential difficulty of software is its conceptual complexity; interfaces are where that complexity is contained. The HVSS ingest defect below was a breaking change shipped as a minor version.
14.2 Tests that protect evidence
Three kinds matter. Acceptance tests state requirements as executable checks: "an event with a field unknown to the schema is stored with that field preserved under extra." Regression tests pin past defects. Contract tests run consumers' expectations against the producer's new version before release. A minimal regression test for the defect below:
def test_unknown_fields_preserved():
raw = {"id": "e1", "rule": "x", "user": "u1", "base": 10, "new_vendor_field": 7}
out = normalise(raw)
assert out["extra"]["new_vendor_field"] == 7
assert sha256(canonical(raw)) == out["raw_sha256"]
Had this test existed before v3.2, the strict schema library would have failed it on the first build. The mfa_reset and grantor_is_self fields would then have stayed under extra instead of disappearing for three weeks.
14.3 Review, deployment and rollback
Code review finds defects that tests miss; inspections were shown early on to remove many defects before testing (Fagan, 1976). Deploy in stages — a canary receiving a copy of traffic, compared with the current version on output counts and field coverage — and keep the previous version deployable. Rollback restores the code, but not data processed meanwhile: if the new version dropped fields, events ingested during the window remain incomplete unless the raw input was kept and can be reprocessed. That is why raw retention (Chapter 4) is a precondition for safe deployment. The Secure Software Development Framework (SP 800-218) lists practices for protecting the build and release pipeline itself.
14.4 Worked example (synthetic): the silent field drop
HVSS's ingest v3.2 switched to a strict schema library that discards unknown keys by default. A vendor had added mfa_reset and grantor_is_self fields two weeks earlier. No test failed, because tests used old fixtures. Three weeks later, a case review found that a self-granted administrator role had been ranked low (Chapter 24 reproduces the effect). Root cause analysis: (1) library default changed behaviour; (2) no acceptance test for unknown-field preservation; (3) fixtures not refreshed from production samples; (4) canary compared counts, not field coverage. Fixes: the regression test above; fixture refresh from sanitised samples monthly; a canary metric "distinct field names per source" with an alert on decrease; and reprocessing of the three-week window from raw storage.
14.5 Release plan
A release plan for ingest changes states: affected interfaces and versions; acceptance tests; canary duration and comparison metrics; rollback trigger and procedure; reprocessing plan; owner. It is a short document, reviewed like code.
For the HVSS v3.2 change, such a plan would have named the strict schema library as the change. It would have compared distinct field names per source during the canary and kept the three-week reprocessing path from raw storage ready before release. Its rollback trigger would have been any drop in distinct field names for the identity-provider source.
14.6 Errors and limits
- Fixtures that never change while inputs do.
- Rollback assumed to undo data effects.
- Schema libraries with lossy defaults.
- Canary metrics that measure volume only.
The HVSS ingest v3.2 incident combines all four errors in a single release. Fixtures written before the vendor added mfa_reset and grantor_is_self could not detect their loss. The library's lossy default discarded them, and a canary that compared event counts saw nothing wrong. Rolling back the code would still not have restored the fields in three weeks of stored events, so reprocessing from raw storage was required.
Exercises — chapter 14
EX027 — Acceptance tests. Objective: encode requirements. Inputs: requirements: (a) unknown fields preserved; (b) events older than 30 days at arrival flagged late but stored; (c) normaliser version recorded. Steps: write three test functions in Python (pseudo-implementation acceptable). Expected evidence: code. Rubric: 3 = three precise tests with boundary cases (exactly 30 days); 2 = two; 1 = vague; 0 = none.
EX028 — Release plan. Objective: plan a safe change. Inputs: a change replacing the timestamp parser. Steps: write a release plan with canary metrics, rollback trigger and reprocessing plan. Expected evidence: plan. Rubric: 3 = includes parsed-timestamp distribution comparison, parse-failure rate, timezone edge cases, raw reprocessing; 2 = partial; 1 = generic; 0 = none.
Chapter synthesis
Response tooling is software whose defects destroy evidence quietly. Version interfaces, test what must be preserved, deploy with canaries that measure information not just volume, keep raw inputs for reprocessing and treat a release plan as a reviewed artefact.
Continue studying — chapter 14 of 50
← Chapter 13 · Course chapter index · Chapter 15 →
Continue to the next authored lesson.