Chapter 6 — Threat intelligence and OSINT: controlled evaluation
Lesson map
Lesson 06 published. Map, example, EX026.
flowchart TD
N0["Learning outcomes"]
N1["6.1 Why evaluate"]
N2["Lab — two triage rules"]
N3["6.2 Reading the result"]
N4["6.3 Which is better?"]
N5["6.4 Fair comparison"]
N6["6.5 Error analysis"]
N7["6.6 Repeating over time"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX026"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["6.1 Why evaluate"]
N2["Lab — two triage rules"]
N3["6.2 Reading the result"]
N4["6.3 Which is better?"]
N5["6.4 Fair comparison"]
N6["6.5 Error analysis"]
N7["6.6 Repeating over time"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX026"]
[Learning outcomes]
|
v
[6.1 Why evaluate]
|
v
[Lab — two triage rules]
|
v
[6.2 Reading the result]
|
v
[6.3 Which is better?]
|
v
[6.4 Fair comparison]
|
v
[6.5 Error analysis]
|
v
[6.6 Repeating over time]
|
v
[Worked example] --> [Exercise EX026]
Learning outcomes
You will (1) evaluate CTI processes with labelled data; (2) compute and interpret precision and recall; (3) design fair comparisons; (4) choose a rule according to consumer cost.
6.1 Why evaluate
Claims such as "our triage rule catches what matters" need evidence. A controlled evaluation applies alternatives to the same labelled set and compares outcomes. At NLG, the question was which triage rule to use for incoming advisories before the operations centre receives them. Rafael's team built a set of synthetic items with labels agreed before either rule was run, then applied both rules to the same set. The result gave the security operations lead evidence for the choice instead of a preference.
Lab — two triage rules
"""Lab (ch 06): compare two triage rules on 20 labelled synthetic items."""
truth = [1,0,0,1,0,1,0,0,0,1,0,0,1,0,0,0,1,0,0,0]
ruleA = [1,1,0,1,0,1,1,0,0,1,0,1,1,0,1,0,1,0,0,0] # broad keyword rule
ruleB = [1,0,0,0,0,1,0,0,0,1,0,0,0,0,0,0,1,0,0,0] # asset-match rule
def pr(p):
tp = sum(a and b for a, b in zip(p, truth)); fp = sum(a and not b for a, b in zip(p, truth))
fn = sum(b and not a for a, b in zip(p, truth))
return tp, fp, fn, round(tp/(tp+fp), 2), round(tp/(tp+fn), 2)
for n, r in (("A", ruleA), ("B", ruleB)):
tp, fp, fn, p, rc = pr(r)
print(f"rule {n}: tp={tp} fp={fp} fn={fn} precision={p} recall={rc}")
Run with python3 lab_eval.py. Verified output:
rule A: tp=6 fp=4 fn=0 precision=0.6 recall=1.0
rule B: tp=4 fp=0 fn=2 precision=1.0 recall=0.67
6.2 Reading the result
On twenty labelled synthetic items, rule A (broad keywords) finds all six relevant items but raises four false alarms: precision 0.6, recall 1.0. Rule B (match against NLG's asset inventory) raises no false alarms but misses two relevant items: precision 1.0, recall 0.67. Rule A wastes analyst time on items that do not matter to NLG, while rule B misses items that do.
In NLG terms, rule A flags ten of the twenty items, and four of those ten would reach the operations centre as advisories about products NLG does not run, which is what precision 0.6 means here. Rule B flags four items, all relevant, but silently drops two that concern a supplier's product.
6.3 Which is better?
Neither in general. If analyst time is scarce and misses can be caught elsewhere, rule B fits. If missing a relevant item is costly — for example, a vulnerability in a customs platform — rule A, or B with a secondary check, fits. The choice depends on costs, not on one metric. For NLG, Rafael recommended rule B plus an inventory update, with a monthly broad-keyword sweep as a safety net. That combination saves analyst time while reducing the risk of missing an item about the customs platform.
6.4 Fair comparison
Use the same items, labels made before seeing rule outputs, and labellers blind to which rule flagged what. Report sample size; twenty items gives wide uncertainty. Twenty items is enough to see the trade-off but too few to estimate precision or recall closely, so Rafael reports counts with each figure. Labels were agreed before the rules ran, and the labeller did not know which rule flagged each item. He records the method with the results, so the evaluation can be repeated with more items.
6.5 Error analysis
Read the misses. Rule B missed two items about a supplier product absent from the inventory — an inventory gap, not a rule flaw. Fixing the inventory may help more than changing the rule. Reading the two misses showed both concerned a supplier product missing from NLG's asset inventory. Rule B could not match what the inventory did not contain. Rafael asks the asset owners to add the product and checks whether other supplier products are missing. The fix improves rule B and every other process that uses the inventory.
6.6 Repeating over time
Threats change; rerun the evaluation quarterly with fresh labelled items. New suppliers, platforms and adversary techniques can change which rule works best. Each quarter, Rafael's team labels a fresh set of synthetic items, applies both rules and compares the results with earlier quarters. If rule B's misses rise, the inventory or the rule needs attention. The results are reported to the security operations lead with counts and the date of each run.
Each quarter's labelled set is kept, so later comparisons can include earlier items as well as fresh ones.
Beyond triage
The same method evaluates other processes: indicator expiry rules, enrichment sources or report templates. Define the outcome, label a sample, apply alternatives and compare. The discipline matters more than the specific metric.
Worked example
Rafael reports: "Rule B saves about four triage reviews per twenty items at the cost of two misses, both caused by inventory gaps. Recommend rule B plus an inventory update, and a monthly broad-keyword sweep as a safety net." The security operations lead agrees.
Synthetic case. The items and rules are invented.
Errors and limits
Labels can be wrong; double-label a sample. Small sets justify only tentative conclusions.
Exercises
- EX026 — Lab metrics. Give precision and recall for both rules. Rubric: 3 = A 0.6/1.0, B 1.0/0.67; 2 = one rule; 1 = wrong; 0 = none.
- EX027 — Cost-based choice. When would rule A be preferable? Rubric: 3 = when misses are costly and analyst time available; 2 = partial; 1 = always; 0 = none.
- EX028 — Blind labelling. Why should labellers not know which rule flagged an item? Rubric: 3 = avoids bias favouring a rule; 2 = partial; 1 = no reason; 0 = none.
- EX029 — Root of misses. What caused rule B's misses and what fix follows? Rubric: 3 = inventory gap; update inventory; 2 = one; 1 = rule bug; 0 = none.
- EX030 — Sample caution. What does a sample of twenty items imply for conclusions? Rubric: 3 = wide uncertainty; tentative; repeat; 2 = partial; 1 = definitive; 0 = none.
Synthesis
Evaluate with labelled data, read both precision and recall, choose by consumer costs and analyse errors before redesigning.
Continue studying — chapter 06 of 12
← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →