Chapter 34 — Threat intelligence and OSINT: integration | Engineering Threat intelligence and OSINT

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 34 — Threat intelligence and OSINT: integration

Lesson map

Lesson 34 published. Map, example, EX100.

flowchart TD
  N0["Learning outcomes"]
  N1["34.1 Integration targets"]
  N2["34.2 Idempotence"]
  N3["34.3 Mapping formats"]
  N4["34.4 Context in tickets"]
  N5["34.5 End-to-end tests"]
  N6["Practice note — owners"]
  N7["Version drift"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX100"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["34.1 Integration targets"]
  N2["34.2 Idempotence"]
  N3["34.3 Mapping formats"]
  N4["34.4 Context in tickets"]
  N5["34.5 End-to-end tests"]
  N6["Practice note — owners"]
  N7["Version drift"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX100"]

[Learning outcomes]
    |
    v
[34.1 Integration targets]
    |
    v
[34.2 Idempotence]
    |
    v
[34.3 Mapping formats]
    |
    v
[34.4 Context in tickets]
    |
    v
[34.5 End-to-end tests]
    |
    v
[Practice note — owners]
    |
    v
[Version drift]
    |
    v
[Worked example] --> [Exercise EX100]

Learning outcomes

You will (1) integrate Lantern with SIEM, DNS, proxy and ticketing systems; (2) design idempotent integrations; (3) map Lantern objects to downstream formats; (4) test integrations end to end.

34.1 Integration targets

SIEM: indicators become correlation lookups; sightings flow back. DNS resolver: domain blocks and response-policy zones. Web proxy: URL categories. Ticketing: high-confidence matches open tickets with Diamond and ATT&CK context.

Each target has its own format, limits and owner, which is why each has its own interface document. Kofi started with the DNS resolver because domain blocks there are precise and easy to reverse. The SIEM came next, then the proxy, then ticketing. Adding targets one at a time meant that problems with one did not mix with problems from another.

34.2 Idempotence

Re-sending the same block list must not duplicate rules. Integrations compare desired state with current state and apply differences only. That also makes rollback a matter of re-applying the previous desired state.

The worked example shows what happens without this: a retry created thousands of duplicate rules. Comparing desired and current state turns a retry into a harmless check. Kofi's integration logs the number of rules added and removed on each run, and a run that reports no changes is the normal, healthy result.

34.3 Mapping formats

STIX patterns such as [domain-name:value = 'c2.example'] translate into each product's rule syntax. Unsupported pattern features are rejected with a reason rather than approximated.

Kofi keeps the translation table for each product in configuration with tests. A pattern that combines two conditions with a time window, for example, cannot be expressed in the resolver's format, so it goes to the SIEM instead. Approximating it with only one condition would block far more than intended or far less. Each rejection is logged with the pattern.

34.4 Context in tickets

A ticket that says "match on drop.example" is weak. A ticket that adds source, confidence, TLP, related ATT&CK techniques and recommended actions lets analysts act quickly.

Kofi measured the effect on a sample of tickets. Analysts closed context-rich tickets faster and escalated fewer of them by mistake, because the recommended action and the confidence were in front of them. A link back to the object in Lantern lets an analyst see the full history without searching. Tickets are generated from the same object.

34.5 End-to-end tests

A synthetic indicator is injected at the collector; the test checks that it appears in the DNS policy, triggers a SIEM lookup on a synthetic query, and opens a ticket — then removes it and checks cleanup.

Kofi runs this test after every release and every downstream change. When the DNS team updated their resolver, the test caught a change in how the policy file was loaded, and the fix was made in staging before production. The synthetic indicator uses a reserved example domain, so even a failed cleanup cannot affect real traffic.

Practice note — owners

Each integration has a named owner on the downstream team. When the DNS team upgrades its resolver, they know to run Lantern's integration test.

Version drift

Downstream products change their APIs. Lantern pins the API version it uses for each integration and runs the end-to-end test in the downstream team's staging environment before they upgrade, so changes surface before production.

Worked example

The proxy integration duplicated 3,000 rules after a retry. Kofi changes it to a state comparison; repeated pushes now report "0 changes", and the proxy team confirms rule counts match Lantern's desired state.

Synthetic case. The duplication incident is invented.

Errors and limits

Downstream products have rule-count limits; prioritise by confidence. Approximating unsupported patterns creates silent gaps.

Exercises

EX100 — Idempotent push. Describe how an idempotent block-list integration works. Rubric: 3 = compare desired vs current, apply diff only; repeat yields no change; 2 = partial; 1 = resend all; 0 = none.
EX101 — Ticket context. List four context fields for a match ticket. Rubric: 3 = four of source, confidence, TLP, ATT&CK, recommended action, Diamond fields; 2 = two; 1 = indicator only; 0 = none.
EX102 — Unsupported pattern. What should happen to a STIX pattern the proxy cannot express? Rubric: 3 = reject with reason/route elsewhere; never approximate silently; 2 = reject; 1 = approximate; 0 = none.

Synthesis

Integrate by desired state, translate patterns exactly, enrich tickets with context and test the whole path.

Continue studying — chapter 34 of 50

← Chapter 33 (draft) · Course chapter index · Chapter 35 (draft) →