◇ OpenCyber Academy
Engineering · Chapter 29 · English

Chapter 29 — Threat intelligence and OSINT: measurement

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 29 — Threat intelligence and OSINT: measurement

Lesson map

Lesson 29 published. Map, example, EX085.

flowchart TD
  N0["Learning outcomes"]
  N1["29.1 Labelled sets"]
  N2["Lab 6 — feed evaluation"]
  N3["29.2 Reading the scores"]
  N4["29.3 Timeliness and uniqueness"]
  N5["29.5 Using results"]
  N6["Sampling for labels"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX085"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["29.1 Labelled sets"]
  N2["Lab 6 — feed evaluation"]
  N3["29.2 Reading the scores"]
  N4["29.3 Timeliness and uniqueness"]
  N5["29.5 Using results"]
  N6["Sampling for labels"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX085"]

[Learning outcomes]
    |
    v
[29.1 Labelled sets]
    |
    v
[Lab 6 — feed evaluation]
    |
    v
[29.2 Reading the scores]
    |
    v
[29.3 Timeliness and uniqueness]
    |
    v
[29.5 Using results]
    |
    v
[Sampling for labels]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX085]

Learning outcomes

You will (1) measure feed quality with precision, recall and F1 against a labelled set; (2) build a labelled evaluation set ethically; (3) measure timeliness and uniqueness; (4) report metrics with their limits.

29.1 Labelled sets

To measure a feed, compare it with ground truth: indicators confirmed malicious in BEN's incidents or in trusted partner investigations, plus confirmed benign values. Labels come from completed investigations, not from the feed itself, to avoid circularity.

Using the feed's own entries as truth would make any feed look perfect. Kofi's labelled set draws on closed investigations where an analyst confirmed whether a value was hostile. Benign labels come from BEN's allow-list of known suppliers and from values investigated and cleared. Each label records who assigned it and when.

Lab 6 — feed evaluation

"""Lab 6 (ch 29): precision and recall of a synthetic feed against a labelled ground-truth set."""
truth_bad = {"a.example", "b.example", "c.example", "d.example", "e.example"}
feed = {"a.example", "b.example", "c.example", "x.example", "y.example", "z.example"}
tp = len(feed & truth_bad); fp = len(feed - truth_bad); fn = len(truth_bad - feed)
p, r = tp / (tp + fp), tp / (tp + fn)
print(f"TP={tp} FP={fp} FN={fn}")
print(f"precision={p:.2f} recall={r:.2f} F1={2*p*r/(p+r):.2f}")

Run with python3 lab_feedeval.py. Verified output:

TP=3 FP=3 FN=2
precision=0.50 recall=0.60 F1=0.55

29.2 Reading the scores

Three of the feed's six entries are confirmed malicious (precision 0.50); it found three of five known malicious values (recall 0.60); F1 is 0.55. Half the feed would be noise if blocked directly.

The scores answer different questions. Precision tells BEN how many blocks from this feed would be mistaken. Recall tells how much of the known hostile activity the feed would have caught. A feed with high recall and low precision can still help as an enrichment source, which is how the worked example ends up using it.

29.3 Timeliness and uniqueness

A feed that lists indicators a week after BEN saw them is of little operational value. Measure median lead time relative to first internal sighting. Uniqueness — indicators no other source provides — shows marginal value.

Kofi measures lead time as the difference between the feed's publication time and BEN's first sighting of the same value. A positive lead means the feed warned BEN in advance; a negative one means it arrived after the fact. Uniqueness is measured against all other sources in Lantern over the same period, since a feed that duplicates others adds little even if its quality is high.

29.4 Limits

Ground truth is incomplete: unknown malicious values count as false positives when they are merely unconfirmed. Report the evaluation period, set size and labelling method with every number.

For that reason Kofi reports precision as a lower bound when unconfirmed entries are counted as false. He also notes how many of the feed's entries could not be labelled at all. With a small set, a single relabelled value can move the scores noticeably, so the report gives the counts behind each ratio. Readers can then judge how much weight the numbers deserve.

29.5 Using results

Metrics feed source weights in Lantern's confidence model and inform renewal decisions for paid feeds. They are shared with partners in aggregate when agreements allow, supporting the reciprocity NIST SP 800-150 describes.

Kofi changes source weights in small steps and records the evaluation that justified each change. A paid feed's renewal depends on the scores over a full contract period, not one quarter. When BEN shares aggregate results with a partner, the partner learns how its feed performs in BEN's environment, which helps it improve without exposing BEN's incident details.

Sampling for labels

Labelling every indicator is impossible. Kofi draws a random sample stratified by indicator type, labels it from investigation records, and reports confidence intervals. A sample of a few hundred, honestly drawn, beats an enormous convenience sample of whatever happened to be investigated.

Worked example

Kofi evaluates a paid feed over a quarter: precision 0.50, recall 0.60, median lead time −2 days (it lags internal sightings). He recommends keeping it as an enrichment source only and not renewing the premium tier.

Synthetic case. Feed values are invented.

Errors and limits

Small labelled sets give unstable estimates. Do not publish named-vendor scores from licensed data.

Exercises

Synthesis

Measure against independent labels, add timeliness and uniqueness, and report every metric with its limits.

Continue studying — chapter 29 of 50

← Chapter 28 (draft) · Course chapter index · Chapter 30 (draft) →