◇ OpenCyber Academy
Technologist · Chapter 23 · English

Chapter 23 — Threat intelligence and OSINT: evaluation

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 23 — Threat intelligence and OSINT: evaluation

Lesson map

Lesson 23 published. Map, example, EX067.

flowchart TD
  N0["Learning outcomes"]
  N1["23.1 What to evaluate"]
  N2["23.2 Source measures"]
  N3["23.3 Product measures"]
  N4["23.4 Comparing fairly"]
  N5["23.5 Recommendations"]
  N6["23.6 Small numbers"]
  N7["23.7 Writing the evaluation"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX067"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["23.1 What to evaluate"]
  N2["23.2 Source measures"]
  N3["23.3 Product measures"]
  N4["23.4 Comparing fairly"]
  N5["23.5 Recommendations"]
  N6["23.6 Small numbers"]
  N7["23.7 Writing the evaluation"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX067"]

[Learning outcomes]
    |
    v
[23.1 What to evaluate]
    |
    v
[23.2 Source measures]
    |
    v
[23.3 Product measures]
    |
    v
[23.4 Comparing fairly]
    |
    v
[23.5 Recommendations]
    |
    v
[23.6 Small numbers]
    |
    v
[23.7 Writing the evaluation]
    |
    v
[Worked example] --> [Exercise EX067]

Learning outcomes

You will (1) evaluate sources and products against requirements; (2) compute simple usefulness measures; (3) gather consumer judgments; (4) recommend keeping, changing or dropping a source.

23.1 What to evaluate

Sources (MCX, vendor list, public advisories) and products (weekly scam list, three-line notes, block requests). Each should serve a requirement.

Linking each source and product to a requirement gives the evaluation its yardstick. MCX serves the requirement to know what is hitting other councils; the weekly scam list serves the service desk's need to answer callers. A source or product that serves no current requirement is a candidate to drop, however interesting its content may be. Lena lists each requirement beside its sources and products at the top of the evaluation, so the reader sees the yardstick first.

23.2 Source measures

For each source over a quarter: items received, items relevant to CMS, items leading to action, false positives reported, median delay from first public appearance. A ratio of actionable to received gives a rough usefulness figure.

Lena's worked example shows why several measures are needed together. MCX sent 120 items with 12 leading to action, a ratio of one in ten; the vendor list sent 2,000 indicators with 6 actions, three in a thousand, plus 9 false-positive complaints. On volume alone the vendor list looks far more productive, which the ratio shows is not the case.

23.3 Product measures

For products: did consumers read them, act on them, find them clear? The monthly feedback form and service-desk call reasons answer these.

A product can be read and still fail if nobody acts on it or if readers misunderstand it. Service-desk call reasons help here: if callers describe a scam that was on Monday's list and the desk handled it without escalation, the list did its job. The feedback form adds what consumers themselves found useful or noisy. Lena also notes when a product was not opened at all, because an unread product cannot be judged on clarity.

23.4 Comparing fairly

Compare sources over the same period and against the same requirements. A source covering a different threat may look worse but fill a gap.

Public advisories, for example, mostly cover software flaws rather than fraud domains, so judging them by blocks produced would be unfair. They meet a different requirement: telling owners about vulnerabilities in products CMS runs. Comparing each source against the requirement it serves avoids dropping the only source that covers a particular need. Lena also uses the same quarter for every source, so seasonal peaks such as a tax deadline affect all of them equally.

23.5 Recommendations

Keep, change (for example, filter by type) or drop. Each recommendation states evidence and cost.

In the worked example, the recommendation for the vendor list is to change rather than drop: filter to domains only, where its false-positive rate was lowest. The evidence is the quarter's counts, and the cost is the licence plus triage time. Stating both lets the manager weigh the recommendation without redoing the analysis. Each recommendation also names who would carry it out, such as the supervisor renegotiating the vendor licence. A recommendation without an owner tends to be agreed in the meeting and then forgotten.

23.6 Small numbers

A quarter may hold few actionable items; avoid strong claims from tiny counts and repeat next quarter.

Six actions from the vendor list in a quarter is a small number, and one more or fewer would move its ratio noticeably. Lena reports counts beside ratios and repeats the evaluation next quarter before treating a pattern as settled. A recommendation based on one quarter is labelled as provisional. The evaluation states the counts behind each ratio in the table, so readers can see for themselves how small the numbers are. Lena keeps the quarterly tables together, so a pattern that holds across several quarters can be told apart from one that appeared once.

23.7 Writing the evaluation

One page: question, method, table, interpretation, recommendation, limits. Managers read the recommendation first.

The question says what is being evaluated, the method shows how, and the table holds the counts, so a reader can check the interpretation against the numbers. Limits come last but are never dropped, for example that context which shaped a decision is not counted as an action. One page keeps the evaluation short enough to be read in a meeting. Lena keeps the detailed counts behind the table, so any figure the manager questions can be traced to its items.

Cost per useful item

Include cost: the vendor list's licence and the analyst hours spent triaging its matches. Dividing total cost by actionable items gives a rough price per useful item. MCX membership costs little, while the vendor list's price per action was high until filtering cut wasted triage. Present this figure carefully, since some value — such as context that shaped a decision — never appears as an action count.

Worked example

Over a quarter MCX delivered 120 items, 30 relevant and 12 actionable; the vendor list delivered 2,000 indicators, 40 matched CMS logs and 6 led to action, with 9 false-positive complaints. Lena recommends keeping MCX, and filtering the vendor list to domains only, where its false-positive rate was lowest.

Synthetic case. The figures are invented.

Errors and limits

Actions are not the only value; context that prevented a wrong decision is hard to count.

Exercises

Synthesis

Evaluate sources and products against requirements with simple measures, compare fairly and recommend with evidence.

Continue studying — chapter 23 of 30

← Chapter 22 (draft) · Course chapter index · Chapter 24 (draft) →