Chapter 4 — Data models | Technologist Incident response and cyber resilience

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 04 published. Map, example, EX010.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["4.1 Entities and relationships"]
  N2["4.2 Fields, types and required values"]
  N3["4.3 Normalization"]
  N4["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4
  N4 --> EX["Worked example"]
  EX --> Q["Exercise EX010"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["4.1 Entities and relationships"]
  N2["4.2 Fields, types and required values"]
  N3["4.3 Normalization"]
  N4["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX010"]

[Learning outcomes and connection]
    |
    v
[4.1 Entities and relationships]
    |
    v
[4.2 Fields, types and required values]
    |
    v
[4.3 Normalization]
    |
    v
[Chapter synthesis]
    |
    v
[Worked example] --> [Exercise EX010]

Learning outcomes and connection

You will be able to (1) design a case record and an event record with typed, required fields; (2) normalize events from different sources into a common schema; and (3) explain how a poor data model produces wrong answers in an investigation. Chapter 3 gave you scripts; scripts need consistent data. The case model you design here is implemented in a tool in Chapter 17 and deployed in Chapter 27.

4.1 Entities and relationships

An incident service handles a few core entities: case (one managed incident or suspected incident), event (a timestamped observation from a source), artefact (a preserved item with a hash), action (something done, by whom, when, with what approval), decision (a choice with its owner and rationale) and asset/identity (what was affected). Relationships: a case has many events, artefacts, actions and decisions; an event references assets and identities. Fernmoor's first spreadsheet had none of this structure: severity was free text and actions had no approver field, so the monthly report could not be trusted. The data model fixes that by giving each entity its own required fields.

4.2 Fields, types and required values

Each field needs a type and a rule. Times: RFC 3339 with offset, stored in UTC. Identifiers: stable keys, not display names (an account id survives a name change). Enumerations: fixed lists for status, severity and source, so that counts mean something. Required fields: what must be known before a record is valid (a case needs an owner and a severity; an action needs an actor and a time). Free text is valuable but should not replace structured fields that you will need to count.

4.3 Normalization

Sources name things differently: user, UserPrincipalName, actor.id. A mapping translates each source's fields into your schema and records the original value. Keep the raw event too: normalization can be wrong, and evidence must remain reviewable. Data-model ideas used by security tools (for example common event schemas published by vendors and open projects) are useful references, but the essential discipline is the same: one meaning per field. For example, the identity provider calls the account userId, while the endpoint agent records a user name; the mapping translates both into the schema's user field and keeps each original.

4.4 Worked example (synthetic): severity counts that lied

Fictional. Fernmoor's first incident spreadsheet had a free-text "Severity" column. Values included "High", "high", "HIGH - maybe", "P1", "sev2?" and blanks. A monthly report counted only exact "High", showing 3 high incidents; a reviewer re-coding by hand found 11. The fix was an enumeration (S1–S4 with definitions from Chapter 13), a required field at triage, and a separate severity_note free-text field for nuance. Historic rows were re-coded by two people independently; disagreements (2 of 64) were resolved and recorded.

A normalized event record adopted at the same time:

event_id      string   required  source-unique id or generated hash
case_id       string   optional  link once triaged
time_utc      RFC3339  required  converted, original kept in raw
source        enum     required  idp | edge | fw | endpoint | app | cloud
actor_id      string   optional  stable account id
asset_id      string   optional  from the inventory (Chapter 11)
action        string   required  normalized verb, e.g. signin, file_read
outcome       enum     required  success | failure | unknown
raw           text     required  original line or JSON

4.5 Common errors and limits

Free-text fields for anything you will later count.
Display names as keys.
Dropping the raw record after normalization.
Converting times without recording the original zone or the correction applied.

Each error damages counting or review. The severity spreadsheet showed what free text does to a monthly report: exact matching on "High" reported 3 high incidents where a hand re-coding found 11. Display names change when people marry or move; stable account ids do not. Keeping the raw record and the original zone lets anyone repeat a conversion and catch a mistake.

Exercises — chapter 4

Attempt each exercise before opening the worked results. Every case, organization, person, address and number in an exercise is synthetic unless a cited public source is named.

EX010 — Case schema. Steps: define ten fields for a case record with type and required/optional. Rubric: 3 = includes id, title, status enum, severity enum, owner, opened/updated times UTC, affected services, links to events/actions, closure reason; 2 = seven sensible fields; 1 = unstructured; 0 = none.

EX011 — Mapping. Steps: map an IdP field set (createdDateTime, userId, ipAddress, status.errorCode) into the normalized event schema from the severity-counts worked example. Rubric: 3 = correct mapping, outcome derived from error code, raw kept, time conversion noted; 2 = one error; 1 = partial; 0 = none.

EX012 — Recode check. Steps: explain why two independent coders were used for historic severity and how to report agreement. Rubric: 3 = reduces individual bias, measure agreement (for example percent agreement 62/64), resolve and record disagreements; 2 = no measure; 1 = vague; 0 = none.

Chapter synthesis

Define entities and fields with types, enumerations and required values; normalize sources into one meaning per field while keeping raw records; and never count free text. A clean case model is the foundation of every later measurement.