Chapter 34 — OT, IoT and cyber-physical security: integration
Lesson map
Lesson 34 published. Map, example, EX100.
flowchart TD
N0["Learning outcomes"]
N1["34.1 Event contract"]
N2["34.2 Idempotent consumers"]
N3["34.3 Contract tests"]
N4["Practice note"]
N5["34.1b Dead letters and schema drills"]
N6["Field note — poison quarantine"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX100"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["34.1 Event contract"]
N2["34.2 Idempotent consumers"]
N3["34.3 Contract tests"]
N4["Practice note"]
N5["34.1b Dead letters and schema drills"]
N6["Field note — poison quarantine"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX100"]
[Learning outcomes]
|
v
[34.1 Event contract]
|
v
[34.2 Idempotent consumers]
|
v
[34.3 Contract tests]
|
v
[Practice note]
|
v
[34.1b Dead letters and schema drills]
|
v
[Field note — poison quarantine]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX100]
Learning outcomes
You will (1) integrate monitor events into the finding store; (2) define idempotent consumers; (3) test contracts with fixtures.
34.1 Event contract
Deny/allow events: schema version, ids, UTC timestamps, source, fc, register, reason, policy version. Consumers ignore unknown fields; producers stay additive.
Lena's event contract at ABP is versioned and published with fixtures for every reason code. Each event carries both the time it happened and the time it was received, plus the clock source. Producers may add fields but never remove or rename them within a version, so older consumers keep working. Fixtures are checked into the repository. Changes are reviewed.
34.2 Idempotent consumers
Finding store keys processed events by event_id. Replaying the log must not create duplicate findings.
Lena tests replay on the bench by feeding a full day's event log through the consumer twice and checking that the finding count does not change. The test caught a bug where events were keyed by timestamp instead of event_id, which duplicated three tickets during a replay storm before the fix landed. Replay tests run in continuous integration. Failures block the merge. The test runs nightly.
34.3 Contract tests
Fixture JSON for each monitor reason code, such as register-out-of-range; consumer tests assert the resulting ticket fields. Run in CI on synthetic fixtures.
Synthetic case — Ashford Batch Plant. A replay storm duplicated three tickets until event_id idempotency landed.
Lena's contract tests run on every pull request and on a schedule. They use one fixture for each reason code and assert the ticket fields the consumer should create, such as asset, technique and severity. A scheduled run against staging catches producer changes even when no pull request is open, so drift is found before a window rather than during one.
34.4 Errors and limits
Clock skew muddies timelines; prefer event time plus receive time.
Lena's timeline view at ABP shows event time and receive time side by side, so a reader can see when a device clock is wrong. Lab II later injected a 90-second skew on PLC-A and found that two events were mis-ordered in the dossier until an offset correction was applied. Clock sources are recorded in each event for exactly this reason. Lena keeps clocks synchronised where possible. Drift is alarmed.
Practice note
Poison events (future timestamps, unknown policy versions) go to a dead-letter queue with an alert — not into findings and not into /dev/null.
34.1b Dead letters and schema drills
Each dead-lettered event keeps the validation error that rejected it. Operators replay after fixing the producer. Once a term, the CNHP team adds an optional field end-to-end in a bench dry run: producer, consumer, fixture, docs. Teams that only evolve schemas in emergencies invent incompatible shortcuts.
A failed scheduled run opens a ticket for the producer team, not the consumer team. Dead-letter age is an SLO chart beside deny-path latency.
ABP runs a schema drill each term. Drill results are recorded with the producer and consumer versions involved, such as the drill that added receive time beside event time on PLC-A's deny events.
Field note — poison quarantine
A producer bug emitted events with timestamps in 2099. Dead-letter quarantine caught them; no findings were created against random hosts. Invisible drops would have been worse — at least quarantine pages someone.
The CNHP consumer canary fails the deploy if poison events from the monitor are not quarantined within one minute in staging.
Each drill ends with the new field visible in a staging ticket for a synthetic PLC-A deny, and the drill's duration recorded.
34.5 Worked example — integrating alerts with the maintenance system (synthetic ABP)
CNHP alerts flow into ABP's maintenance-management system so that a "program download to PLC-A" alert can be matched to an open work order. The event contract carries an event ID, asset tag, ATT&CK for ICS technique (T0843 Program Download), time with clock source, and evidence hash. The consumer is idempotent on event ID, so a replayed alert does not create two work orders. Contract tests run in CI with fixtures from both sides; an event lacking a technique field is sent to a dead-letter queue and reviewed daily, rather than dropped. A download with no matching work order escalates to the shift lead within five minutes.
Exercises
- EX100 — Field. Why include policy_version on every event? Rubric: 3 = audit which rules applied; 2 = vague; 1 = wrong; 0 = none.
- EX101 — Replay. How does event_id prevent duplicate findings? Rubric: 3 = upsert/ignore duplicates; 2 = vague; 1 = wrong; 0 = none.
- EX102 — Fixture. Two fixtures required before a new reason code. Rubric: 3 = producer example + consumer expected ticket; 2 = one; 1 = vague; 0 = none.
Synthesis
Contracts, idempotent consumers, fixture tests.
Continue studying — chapter 34 of 50
← Chapter 33 (draft) · Course chapter index · Chapter 35 (draft) →