Chapter 41 — Threat intelligence and OSINT: system optimization
Lesson map
Lesson 41 published. Map, example, EX121.
flowchart TD
N0["Learning outcomes"]
N1["41.1 Outcome first"]
N2["41.2 Binding constraint"]
N3["41.3 Candidate optimisations"]
N4["41.4 Measuring the effect"]
N5["41.5 Goodhart's warning"]
N6["Practice note — removing work"]
N7["Cost of complexity"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX121"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["41.1 Outcome first"]
N2["41.2 Binding constraint"]
N3["41.3 Candidate optimisations"]
N4["41.4 Measuring the effect"]
N5["41.5 Goodhart's warning"]
N6["Practice note — removing work"]
N7["Cost of complexity"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX121"]
[Learning outcomes]
|
v
[41.1 Outcome first]
|
v
[41.2 Binding constraint]
|
v
[41.3 Candidate optimisations]
|
v
[41.4 Measuring the effect]
|
v
[41.5 Goodhart's warning]
|
v
[Practice note — removing work]
|
v
[Cost of complexity]
|
v
[Worked example] --> [Exercise EX121]
Learning outcomes
You will (1) optimise Lantern for the outcome that matters — timely, accurate enforcement; (2) identify the binding constraint across people and machines; (3) evaluate optimisations with before-and-after measurement; (4) avoid optimising proxies that harm outcomes.
41.1 Outcome first
The target outcome is fewer successful attacks reaching BEN, approximated by time-to-enforcement for high-confidence indicators and the share of incidents with intelligence context at triage. Optimising "objects ingested per day" can make outcomes worse by flooding analysts.
Kofi states the outcome before proposing any change, and asks how the change would move it. A faster parser that lets more low-confidence indicators through can raise volume while slowing analysts down. Both proxy measures have known weaknesses, and he reports them together so that improving one at the other's expense is visible.
41.2 Binding constraint
Often the slowest step is human: analyst approval for sharing, OT engineer review. Faster machines do not help if approvals wait overnight. Optimisation may mean better prioritisation of approval queues rather than new hardware.
Kofi traced a sample of indicators from arrival to enforcement and timed each step. Machine stages took minutes in total; the wait for a person to approve sharing took hours. That single measurement redirected the optimisation effort away from the servers and towards how the approval queue was organised, which is the subject of the worked example.
41.3 Candidate optimisations
Pre-filter low-value indicators before enrichment; prioritise approval queues by confidence and asset criticality; batch firewall pushes on a fixed cadence; reduce duplicate sources whose uniqueness measured near zero.
Kofi listed each candidate with its expected effect, cost and risk before starting any of them. Pre-filtering saves enrichment capacity but risks discarding something useful, so the filter rule was agreed with analysts. Batching pushes reduces load on firewalls but adds a little delay. Dropping duplicate sources needed the uniqueness figures from the measurement work, which showed two feeds adding almost nothing that others did not already supply.
41.4 Measuring the effect
Each change has a hypothesis, a metric and a comparison window. If the metric does not move, the change is reverted to avoid complexity without benefit.
Kofi writes the hypothesis down before the change, such as: sorting the approval queue by confidence will cut median wait by half within a month. The comparison window covers the same weekdays before and after, since volumes differ through the week. One change he tried, caching a rarely used lookup, made no measurable difference and was removed. Reverting is treated as a normal outcome, not a failure.
41.5 Goodhart's warning
When a measure becomes a target, people game it. If analysts are judged on approvals per hour, they may approve poorly. Keep quality checks beside speed metrics.
The worked example guards against this with a spot check of retractions after the approval process sped up. If faster approvals had produced more mistakes, the gain would have been false. Kofi pairs every speed metric with a quality metric on the same dashboard, so nobody sees one without the other. Analysts are not ranked on either; the figures describe the process, not individuals.
Practice note — removing work
The best optimisation is often deletion: dropping a feed that adds nothing unique removes ingestion, storage and analyst noise at once.
Cost of complexity
Every optimisation adds code or process to maintain. Kofi estimates the yearly maintenance hours of each change and compares them with the hours it saves analysts.
Worked example
Analyst approval of indicators for sharing has a median wait of 6 hours. Kofi sorts the queue by confidence and partner demand and adds a daily 30-minute approval slot. Median wait falls to 1.5 hours; a spot check shows no rise in later retractions.
Synthetic case. Figures are invented.
Errors and limits
Local optimisations can move bottlenecks elsewhere; re-measure the whole path. Some delays are deliberate safety controls and must not be optimised away.
Exercises
- EX121 — Proxy trap. Why can maximising objects ingested per day harm outcomes? Rubric: 3 = more noise/analyst overload/lower precision; 2 = partial; 1 = always good; 0 = none.
- EX122 — Quality check. What quality check accompanied the approval-speed improvement? Rubric: 3 = spot check showed no rise in retractions; 2 = some check; 1 = vague; 0 = none.
- EX123 — Deliberate delay. Name one delay in Lantern that must not be optimised away. Rubric: 3 = OT two-person approval (or similar safety gate) with reason; 2 = name only; 1 = vague; 0 = none.
Synthesis
Optimise outcomes, find the real constraint, measure every change, guard against Goodhart and keep safety delays.
Continue studying — chapter 41 of 50
← Chapter 40 (draft) · Course chapter index · Chapter 42 (draft) →