Chapter 34 — Authorized offensive security and red team: integration | Engineering Authorized offensive security and red team

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 34 — Authorized offensive security and red team: integration

Lesson map

Lesson 34 published. Map, example, EX100.

flowchart TD
  N0["Learning outcomes"]
  N1["34.1 Event contract"]
  N2["34.2 Idempotent consumers"]
  N3["34.3 Contract tests"]
  N4["34.2b Poison events"]
  N5["34.3b Schema evolution drill"]
  N6["34.1b Dead-letter queues"]
  N7["Practice note"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX100"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["34.1 Event contract"]
  N2["34.2 Idempotent consumers"]
  N3["34.3 Contract tests"]
  N4["34.2b Poison events"]
  N5["34.3b Schema evolution drill"]
  N6["34.1b Dead-letter queues"]
  N7["Practice note"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX100"]

[Learning outcomes]
    |
    v
[34.1 Event contract]
    |
    v
[34.2 Idempotent consumers]
    |
    v
[34.3 Contract tests]
    |
    v
[34.2b Poison events]
    |
    v
[34.3b Schema evolution drill]
    |
    v
[34.1b Dead-letter queues]
    |
    v
[Practice note]
    |
    v
[Worked example] --> [Exercise EX100]

Learning outcomes

You will (1) integrate TrackScope events into FindingsDesk; (2) define idempotent consumers; (3) test contracts with fixtures.

34.1 Event contract

Deny/allow events: schema version, ids, timestamps (UTC), targets, reasons. Consumers ignore unknown fields; producers stay additive.

UTC timestamps and additive schemas keep consumers compatible. Ignoring unknown fields is required so producers can evolve. Sam publishes the contract in a repo both teams own, with examples for each reason code. Examples are golden fixtures; changing them needs the contract owner's review. When a producer and an example disagree, deploys pause until the pack matches — the same discipline that stopped the replay storm from minting duplicate tickets after event_id idempotency landed.

34.2 Idempotent consumers

FindingsDesk keys processed events by event_id. Replaying the log must not create duplicate findings.

event_id keys turn replay from a duplication bug into a recovery tool. The three duplicated tickets in the storm story are the cautionary tale. Sam's consumer suite includes a double-delivery fixture for every event type. Double-delivery fixtures assert ticket counts stay at one after replay, counting through the findings API rather than grepping logs, because log rotation during the replay storm would have hidden the three duplicates that first exposed the bug.

34.3 Contract tests

Fixture JSON for each reason code; consumer tests assert resulting ticket fields. Run in CI on synthetic fixtures.

Synthetic case — Riverstone Transit. A replay storm duplicated three tickets until event_id idempotency landed.

Fixtures per reason code lock ticket fields to the contract. CI on synthetic JSON catches breaks before engagements. Sam requires PulseMap and FindingsDesk both consume the same fixture pack. Shared fixture packs live in the contract repo, not copied into each service. CI hashes the pack and fails any service tree that contains a divergent copy, which is how PulseMap and FindingsDesk stayed on the same reason-code fixtures after the catalogue gained resume-expired.

34.4 Errors and limits

Clock skew between hosts muddies timelines; prefer event time + receive time.

Event time plus receive time explains skew without blaming one host's clock. Sam's timeline UI always shows both, and analysts are trained to sort by event time while watching receive time for transport delay when reconstructing a stop. Sorting guidance is printed in the analyst handbook beside a synthetic skew screenshot labelled as a teaching aid. When the timeline UI changed last term, the screenshot was updated in the same pull request so analysts would not learn a layout that no longer existed.

34.2b Poison events

Consumers must validate schema and reject events with future timestamps, unknown policy versions, or targets that fail format checks. A poisoned event stream should quarantine, not create findings against random hosts.

Quarantine beats creating findings against garbage hosts. Sam pages on quarantine growth the same way he pages on deny spikes. Quarantine pages include the producing service name. The rollback runbook opens on that name from the page payload, so an operator facing a poisoned stream of future-timestamped events can stop the producer without guessing which service to kill.

34.3b Schema evolution drill

Practice adding an optional field end-to-end in a dry run: producer, consumer, fixture, docs. Teams that only evolve schemas in emergencies invent incompatible shortcuts.

End-to-end optional fields keep muscles memory honest. Sam schedules a drill each term with a throwaway field name. Throwaway fields are deleted at term end with a migration note that names the owning team. That team must acknowledge the note in the contract repo before the field vanishes, so a dry-run field cannot disappear while a consumer still reads it.

34.1b Dead-letter queues

Events that fail validation go to a dead-letter queue with an alert — not to /dev/null and not into findings. Operators replay after fixing the producer. Invisible drops recreate repudiation.

Sam's runbook forbids filtering alerts on that queue without an AUTH note. AUTH notes for alert filters expire after seven days unless renewed with the same dual approval as the original note. Expired filters re-arm automatically; that rule caught a forgotten filter that had silenced dead-letter pages for nine days during a quiet exam period.

Practice note

Contract tests run on a schedule against staging fixtures even when no PR is open, catching silent producer drift. A green CI on Monday and a broken consumer on Thursday usually means someone edited a fixture by hand in an environment CI does not see. Scheduled contract tests close that gap.

Include a consumer canary that fails the deploy if poison events are not quarantined within one minute in staging.

Dead-letter age is an SLO chart beside deny-path latency.

34.4b Ownership of the event contract

Each event type on the bus has one owning team that approves changes. PulseMap consumes FindingsDesk events but cannot alter their schema; it files a request instead, which keeps the contract stable for every other consumer.

Approvals are recorded in the contract repo. Sam blocks merges that touch a type without the owner team's review. Owner team aliases are group addresses such as trackscope-contract@riverstone.example, so ownership survives when individuals rotate off the platform squad and every schema change still reaches someone on call.

Exercises

EX100 — Field. Why include policy_version on every event? Rubric: 3 = audit which ROE applied; 2 = vague; 1 = wrong; 0 = none.
EX101 — Replay. How does event_id prevent duplicate findings? Rubric: 3 = upsert/ignore duplicates; 2 = vague; 1 = wrong; 0 = none.
EX102 — Fixture. Name two fixtures required before integrating a new reason code. Rubric: 3 = producer example + consumer expected ticket; 2 = one; 1 = vague; 0 = none.

Synthesis

Contracts, idempotent consumers, fixture tests.

Continue studying — chapter 34 of 50

← Chapter 33 (draft) · Course chapter index · Chapter 35 (draft) →