Chapter 22 — Security operations and blue team: system and data models

Lesson map

Lesson 22 published. Map, example, EX064.

flowchart TD
  N0["Learning outcomes"]
  N1["22.1 Pipeline model"]
  N2["22.2 Event model"]
  N3["22.3 Entity model"]
  N4["22.4 Relationships"]
  N5["22.5 Gaps from models"]
  N6["22.6 Model ownership"]
  N7["22.7 Models and people"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX064"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["22.1 Pipeline model"]
  N2["22.2 Event model"]
  N3["22.3 Entity model"]
  N4["22.4 Relationships"]
  N5["22.5 Gaps from models"]
  N6["22.6 Model ownership"]
  N7["22.7 Models and people"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX064"]

[Learning outcomes]
    |
    v
[22.1 Pipeline model]
    |
    v
[22.2 Event model]
    |
    v
[22.3 Entity model]
    |
    v
[22.4 Relationships]
    |
    v
[22.5 Gaps from models]
    |
    v
[22.6 Model ownership]
    |
    v
[22.7 Models and people]
    |
    v
[Worked example] --> [Exercise EX064]

Learning outcomes

You will (1) model the SOC pipeline as stages and data stores; (2) define the event data model; (3) model entities such as users, hosts and customers; (4) use models to find gaps.

22.1 Pipeline model

Lindqvist's pipeline has stages: collect (agents, syslog, cloud APIs), transport, parse and normalise, enrich (asset and identity data), store, detect, alert, and case management. Each stage can fail, delay or distort data.

Thinking in stages helps locate problems. A missing alert could mean the event was never collected, was dropped in transport, failed to parse, lacked enrichment the rule needed, aged out of storage, or never matched the rule. Each stage gets its own health measures, so an analyst can trace an event through the pipeline and find where it disappeared.

22.2 Event model

Every normalised event carries time (UTC), source, host, user, action, outcome, and raw record reference. A shared model lets one rule work across sources.

Keeping a reference to the raw record matters because normalisation can lose detail or introduce errors; an analyst may need the original line to check what the source actually said. Recording time in UTC avoids confusion between sources in different zones. Because every source maps to the same fields, a rule about failed logins works across the portal, VPN and staff sign-in.

22.3 Entity model

Entities persist across events: a customer, an employee, a host, a network. Entity tables hold attributes such as criticality, owner and normal locations, used for enrichment.

Events are moments; entities give them context. An alert about a host means more when the alert also shows that the host is a payment server owned by finance. Normal locations for a customer help judge whether a login from a new country is unusual. Entity tables need updating, since stale data misleads analysts as surely as missing data.

22.4 Relationships

Customers own accounts; employees use hosts; hosts sit in zones. Modelling relationships lets analysts ask "which zone is this host in?" automatically.

Without modelled relationships, an analyst must look up each link manually in different systems, which is slow during an incident. With them, an alert can show automatically that the host belongs to a depot zone and that its usual user is on leave. Relationships also let rules reason across entities, for example flagging an employee using a host outside their own zone.

22.5 Gaps from models

Walking through the model stage by stage, Amara checks each source: is it collected, parsed, enriched and covered by detections? Depot networks are collected but not enriched, because the asset inventory lacks depot devices.

The walk-through is systematic: for each source, four yes-or-no questions. Missing enrichment for depot devices means alerts about them carry only an address, with no owner, location or role, so analysts must investigate from scratch. The fix is upstream: adding depot devices to the asset inventory, which helps IT operations as well as the SOC.

22.6 Model ownership

The data model is owned by the detection engineering team; changes are versioned like rules.

An owner decides which fields exist and what they mean, and reviews proposed changes. Versioning lets rules declare which model version they expect, so a field rename does not silently break them. Without ownership, different engineers add fields with overlapping meanings, and over time the shared model splinters into source-specific dialects. The owner publishes a changelog, and rule authors read it before each release. Deprecated fields stay available for one release so rules can migrate without a gap in coverage.

22.7 Models and people

Models also include who acts at each stage: which team fixes parsing errors, who maintains enrichment tables.

A pipeline diagram that shows only systems hides the human work that keeps it running. Naming the team responsible for parser failures means alerts about broken parsing reach someone who can act. Naming the owner of enrichment tables means stale entries have someone accountable. Amara adds an owner to every stage on the pipeline diagram. Each owner also has a backup named, so a parser failure during holidays still reaches someone able to fix it.

Enrichment freshness

Enrichment tables must be current. A host reassigned from finance to depots still tagged 'finance' misleads analysts. Amara adds a freshness date to each table.

Worked example

Amara's coverage matrix lists 14 sources against 6 stages. Two cells are empty for depot networks (enrichment, detection) and one for the portal load balancer (detection, newly collected). She turns each empty cell into a backlog item with an owner.

Synthetic case. Sources and matrix are invented.

Errors and limits

Models simplify; a source marked "collected" may still have gaps in time or fields.

Exercises

• EX064 — Pipeline stages. List the pipeline stages in order.
  Show marking criteria

  Rubric: 3 = collect, transport, parse/normalise, enrich, store, detect, alert, case; 2 = most in order; 1 = few; 0 = none.
• EX065 — Depot gap. Why were depot networks not enriched?
  Show marking criteria

  Rubric: 3 = asset inventory lacked depot devices; 2 = partial; 1 = no reason; 0 = none.
• EX066 — Matrix use. How did Amara use the coverage matrix?
  Show marking criteria

  Rubric: 3 = empty cells became owned backlog items; 2 = partial; 1 = filed it; 0 = none.

Synthesis

Pipeline, event and entity models expose where data is missing, unenriched or undetected.

Continue studying — chapter 22 of 50

← Chapter 21 (draft) · Course chapter index · Chapter 23 (draft) →