Chapter 22 — Incident data, telemetry and provenance models | Engineering Incident response and cyber resilience

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 22 published. Map, example, EX043.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["22.1 Schema elements"]
  N2["22.2 Entity resolution"]
  N3["22.3 Lineage"]
  N4["22.4 Retention and integrity"]
  N5["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX043"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["22.1 Schema elements"]
  N2["22.2 Entity resolution"]
  N3["22.3 Lineage"]
  N4["22.4 Retention and integrity"]
  N5["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX043"]

[Learning outcomes and connection]
    |
    v
[22.1 Schema elements]
    |
    v
[22.2 Entity resolution]
    |
    v
[22.3 Lineage]
    |
    v
[22.4 Retention and integrity]
    |
    v
[Chapter synthesis]
    |
    v
[Worked example] --> [Exercise EX043]

Learning outcomes and connection

After this chapter you can (1) design an event schema with entity keys, multiple timestamps and lineage; (2) write normalisation rules for mismatched identifiers across identity, host and cloud sources; (3) trace a case decision back to the raw records that supported it; and (4) set retention and integrity rules per data class. It builds on Chapters 9, 11 and 13 and prepares Chapters 24, 28 and 34.

22.1 Schema elements

A response event schema needs at least: an event ID; timestamps — event time (when it happened, per the source), observed time (when the source recorded it), ingest time (when we received it) and the source's clock offset if known; entity references — user, host, process, resource — with the identifier as given and the resolved canonical entity; source and collector; raw reference (hash and storage pointer); and lineage — the chain of transformations with versions. Keeping event and ingest time separate makes delays measurable (Chapter 36) and lets rules evaluate on event time.

22.2 Entity resolution

The same person appears as j.ortiz@hvss.example in the identity provider, HVSS\jortiz on Windows hosts and an opaque object ID in the cloud audit log. Resolution maps each to a canonical entity using authoritative directories, with a record of which mapping rule applied and when the directory was read. Resolution is evidence-relevant: a wrong mapping attributes actions to the wrong person. Keep the original identifier alongside the resolved one, never replace it. In case C-2207, the cloud actor 9f2… was resolved to person:0193 using the 06:00 directory snapshot.

22.3 Lineage

Lineage answers "where did this value come from?" The W3C PROV model represents entities, activities and agents; in practice, store for each derived record the IDs of its inputs and the activity (normaliser version, enrichment source, rule version). A case decision then references derived records, which reference raw records — a traversable graph.

The C-2207 trace in the worked example is this graph in practice. The decision record points to D-881 and D-884, and these point to raw events R-55102, R-55140 and R-55171. Each link names the normaliser version and the directory snapshot used, so a reviewer can follow the chain without asking the analyst.

22.4 Retention and integrity

Different classes need different rules: raw security telemetry (integrity-critical, long retention, hash manifest); derived events (reproducible from raw, shorter retention acceptable); case notes (records-management rules); personal data (minimisation, legal basis). The FAIR principles — findable, accessible, interoperable, reusable — written for research data, are a useful checklist for whether evidence will be usable later. For case C-2207, the IdP and cloud audit raw events fall in the first class and were checked against the manifest anchored at 00:00, while D-881 and D-884 could be regenerated from them by normaliser v4.1.

22.5 Worked example (synthetic): tracing a decision at HVSS

Decision: "disable account j.ortiz at 14:20". Trace:

Case C-2207 decision record → references derived events D-881 (impossible-travel alert) and D-884 (new mail-forwarding rule).
D-881 ← IdP sign-in raw events R-55102 (Lisbon, 13:58 event time) and R-55140 (synthetic remote address, 14:03), resolved to entity person:0193 by directory snapshot of 06:00; normaliser v4.1.
D-884 ← cloud audit raw event R-55171, actor object ID 9f2… resolved to person:0193 by the same snapshot.
Raw events verified against the daily manifest anchored at 00:00.

The trace revealed a weakness: the directory snapshot was eight hours old; had the account been renamed that morning, resolution could have been wrong. Rule added: resolution records the snapshot time, and decisions on accounts changed after the snapshot require a live lookup.

22.6 Errors and limits

One timestamp per event.
Overwriting source identifiers with resolved ones.
Lineage as free-text notes.
One retention rule for all data.

The C-2207 trace in the worked example works only because each error was avoided. With one timestamp per event, the Lisbon sign-in at 13:58 could not be compared with its ingest time to measure delivery delay. If 9f2… had been overwritten by person:0193, the reviewer could not have checked the eight-hour-old directory mapping. Free-text lineage would not let the decision be traversed back to R-55171. One retention rule would also force the derived events and the raw events to share a lifetime they do not need.

Exercises — chapter 22

EX043 — Normalisation rules. Objective: specify resolution. Inputs: three synthetic identifiers for one person (UPN, DOMAIN\sam, cloud object ID) and a contractor whose UPN was reused from a former employee. Steps: write resolution rules with precedence and validity periods; show how the reuse case is handled. Expected evidence: rules and a worked case. Rubric: 3 = immutable IDs preferred, validity intervals, reuse handled by time-bounded mapping; 2 = partial; 1 = string matching; 0 = none.

EX044 — Schema. Objective: design fields. Inputs: the schema elements in 22.1. Steps: write a JSON example of one normalised event with all elements and two lineage steps. Expected evidence: JSON. Rubric: 3 = four timestamps, original and resolved IDs, raw hash, lineage with versions; 2 = missing one; 1 = minimal; 0 = none.

Chapter synthesis

A response data model keeps multiple timestamps, original and resolved identities, raw references and versioned lineage, with retention and integrity rules per data class, so that every decision can be traced to the records that supported it.