OpenCyber Academy
Engineering · Chapter 22 · English
Course code: OCA-133Chapter 22: OCA-133-C022

Chapter 22 — Threat intelligence and OSINT: system and data models

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 22 published. Map, example, EX064.

flowchart TD
  N0["Learning outcomes"]
  N1["22.1 STIX 2.1 core"]
  N2["22.2 Choosing a subset"]
  N3["22.3 Confidence and markings"]
  N4["22.4 Validation"]
  N5["22.5 System model"]
  N6["Practice note — Mapping external references"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX064"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["22.1 STIX 2.1 core"]
  N2["22.2 Choosing a subset"]
  N3["22.3 Confidence and markings"]
  N4["22.4 Validation"]
  N5["22.5 System model"]
  N6["Practice note — Mapping external references"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX064"]

[Learning outcomes]
    |
    v
[22.1 STIX 2.1 core]
    |
    v
[22.2 Choosing a subset]
    |
    v
[22.3 Confidence and markings]
    |
    v
[22.4 Validation]
    |
    v
[22.5 System model]
    |
    v
[Practice note — Mapping external references]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX064]

Learning outcomes

You will (1) model Lantern's data with STIX 2.1 objects and relationships; (2) choose which STIX objects to support; (3) model provenance and confidence; (4) validate objects against required properties.

22.1 STIX 2.1 core

STIX 2.1 defines domain objects (indicator, malware, threat-actor, campaign, attack-pattern, report and others), cyber-observable objects (domain-name, ipv4-addr, file), and relationship and sighting objects. Every domain object needs type, spec_version, id, created and modified; indicators also need pattern, pattern_type and valid_from.

Lantern's validator checks these required properties on every object before anything else. Indicators are the type most partners send, and their pattern field holds the condition to match, such as a domain value. Kofi keeps a short internal guide with examples of valid objects for each supported type, which partners have found useful when debugging their own exports.

22.2 Choosing a subset

Lantern supports indicator, attack-pattern (linked to ATT&CK IDs via external references), campaign, threat-actor, malware, report, relationship and sighting. Unsupported types are stored opaquely and flagged for review rather than dropped.

Kofi chose the subset by counting what partners actually sent over three months and what analysts actually queried. Types that appeared rarely and that nobody searched for were left out, which saved building views and validation rules for them. Storing unsupported objects as opaque records means nothing is lost if a later need arises, and the review flag shows how often each unsupported type arrives.

22.3 Confidence and markings

STIX offers a confidence property (0–100) and marking definitions, including TLP markings. Lantern maps its internal decayed confidence to that property on export and attaches TLP 2.0 markings to every shared object.

Because each producer sets confidence its own way, Lantern records the value a partner sent alongside its own score. Analysts can then see when BEN's view differs from the source's. On export, the TLP marking is chosen from the strictest source label, and an object with no label from its source is not shared at all until an analyst assigns one.

22.4 Validation

Before storing, the validator checks required properties and identifier format. The masters-level lab in this course family checks the same rules; Lantern enforces them in the pipeline and counts failures per source.

The per-source failure counts feed the partner data-quality reports. Kofi found that most failures came from two partners, both sending timestamps without time zones. A short note to each partner, with examples, fixed the problem within a month. The validator rejects rather than repairs, because a repaired timestamp would rest on a guess about the partner's intended zone.

22.5 System model

Beyond data: components (collector, validator, store, enricher, analyst UI, publisher), their interfaces and states. A state machine for an indicator — new, active, decayed, retired — makes behaviour explicit.

The state machine prevents odd combinations, such as a retired indicator still being pushed to firewalls. Kofi listed each allowed transition with its trigger: a new object becomes active after validation, decays as time passes without sightings and is retired when it crosses the floor. A sighting can move a decayed indicator back to active. Any transition outside the list is an error and is logged.

Practice note — Mapping external references

Attack-pattern objects carry external references to ATT&CK technique pages, for example T1566 Phishing. Lantern validates that such references use real technique IDs, because mistyped IDs silently break detection coverage reports that group findings by technique.

Worked example

A partner sends a threat-actor object without spec_version. Lantern rejects it with a reason, stores the hash, and notifies the partner's data contact. Their next export includes the property.

Synthetic case. The object and partner are invented.

Errors and limits

STIX is expressive; supporting everything adds complexity without value. Confidence numbers from different producers are not directly comparable.

Exercises

Synthesis

Use a deliberate STIX subset, validate required properties, carry confidence and TLP markings, and make states explicit.

Continue studying — chapter 22 of 50

← Chapter 21 (draft) · Course chapter index · Chapter 23 (draft) →

Worked answers

EX064. pattern, pattern_type and valid_from.

EX065. Store it opaquely with provenance and flag it for review, so nothing is silently lost.

EX066. new → active → decayed → retired. Example triggers: active → decayed as time passes without sightings; decayed → retired when confidence falls below the floor; decayed → active on a new sighting.