Chapter 28 — Threat intelligence and OSINT: implementation I
Lesson map
Lesson 28 published. Map, example, EX082.
flowchart TD
N0["Learning outcomes"]
N1["28.1 Client stage"]
N2["28.2 Validation and normalisation"]
N3["28.3 Queues"]
N4["28.4 Metrics from day one"]
N5["28.5 Synthetic server stub"]
N6["Configuration of partners"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX082"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["28.1 Client stage"]
N2["28.2 Validation and normalisation"]
N3["28.3 Queues"]
N4["28.4 Metrics from day one"]
N5["28.5 Synthetic server stub"]
N6["Configuration of partners"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX082"]
[Learning outcomes]
|
v
[28.1 Client stage]
|
v
[28.2 Validation and normalisation]
|
v
[28.3 Queues]
|
v
[28.4 Metrics from day one]
|
v
[28.5 Synthetic server stub]
|
v
[Configuration of partners]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX082]
Learning outcomes
You will (1) implement a TAXII 2.1 client stage against a synthetic server stub; (2) implement validation and normalisation stages; (3) wire stages with bounded queues; (4) produce operator-visible metrics from the first release.
28.1 Client stage
The client discovers the API root, lists permitted collections, and requests objects added after the last saved checkpoint. It follows pagination until complete and stores the new checkpoint only after the batch is safely written, so a crash causes a re-fetch rather than a gap.
The order matters, as the worked example shows. A checkpoint saved before writing marks objects as received when they are not. Saving after writing means a crash between the two steps fetches some objects twice, which the store handles because writes are keyed by identifier and repeated writes change nothing.
28.2 Validation and normalisation
Validation enforces STIX required properties and identifier format. Normalisation applies the documented rules from Chapter 13. Both stages emit rejects with reasons.
Running validation before normalisation means that rules about formats are checked on what the partner actually sent. Normalising first could hide a malformed value by turning it into something that looks valid. Kofi's stage logs show counts per source and per reason, which become the partner quality reports. A record rejected here never reaches the store, so an analyst never sees a half-cleaned value.
28.3 Queues
Bounded queues connect stages. If enrichment slows, validation fills its output queue and waits; the client pauses. This protects memory and makes slowness visible.
Kofi chose queue sizes from measured memory use per object and a limit that leaves room for the rest of the host. When a queue stays full for more than a few minutes, an alert tells the operator which stage is slow. Unbounded queues would hide the slowness until memory ran out and the whole collector stopped at once.
28.4 Metrics from day one
Objects received, accepted, rejected per source; queue depths; checkpoint age; poll duration. Metrics are cheap to add early and expensive to retrofit.
Kofi added these counters in the first week, before any performance work. They showed within days that one partner's polls took far longer than the others, which turned out to be a slow server on the partner side, not a problem in Lantern. Without the poll duration metric the team might have spent days profiling the wrong code. Each metric has a short description and an owner on the dashboard.
28.5 Synthetic server stub
Development uses a local stub that serves synthetic STIX bundles in TAXII 2.1 form. No development traffic touches partner servers, protecting partners and keeping tests reproducible.
The stub serves fixed bundles from files, including malformed ones for error tests, and can be told to respond slowly or refuse requests. That lets the team test timeouts and back-off without asking a partner for a broken server. Because the bundles are fixed, a test that passes today will pass tomorrow unless the code changed.
Configuration of partners
Each partner entry in configuration names the API root, collections, polling interval, credential reference and data contact. Adding a partner is a reviewed configuration change, not a code change, which keeps the client stage generic and makes the partner list itself auditable. Removing a partner follows the retirement steps from Chapter 16.
Worked example
On the first test run the client saved its checkpoint before writing the batch; a forced crash then lost 200 objects. Kofi reorders the steps — write, then checkpoint — and adds a test that crashes between them and verifies no loss.
Synthetic case. The bug and counts are invented.
Errors and limits
Exactly-once delivery is hard; design for at-least-once plus idempotent writes. Testing against live partner servers is not acceptable for development.
Exercises
- EX082 — Checkpoint order. Why must the checkpoint be saved after the batch is written? Rubric: 3 = otherwise crash loses objects; re-fetch is safe with idempotent writes; 2 = partial; 1 = order irrelevant; 0 = none.
- EX083 — Stub reason. Give two reasons to develop against a synthetic TAXII stub. Rubric: 3 = no load/risk to partners + reproducible tests; 2 = one; 1 = faster only; 0 = none.
- EX084 — Early metric. Which metric reveals a stalled collector, and why? Rubric: 3 = checkpoint age / objects received per source; rises or drops to zero when stalled; 2 = metric only; 1 = CPU; 0 = none.
Synthesis
Checkpoint after writing, validate and normalise with rejects, bound the queues, and ship metrics with the first release.
Continue studying — chapter 28 of 50
← Chapter 27 (draft) · Course chapter index · Chapter 29 (draft) →