Chapter 36 — Threat intelligence and OSINT: performance and scale
Lesson map
Lesson 36 published. Map, example, EX106.
flowchart TD
N0["Learning outcomes"]
N1["36.1 Unstable queues"]
N2["Lab 8 — queue growth"]
N3["36.2 Little's law"]
N4["36.3 Options"]
N5["36.4 Protecting partners"]
N6["36.5 Testing at scale"]
N7["Practice note — priority queues"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX106"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["36.1 Unstable queues"]
N2["Lab 8 — queue growth"]
N3["36.2 Little's law"]
N4["36.3 Options"]
N5["36.4 Protecting partners"]
N6["36.5 Testing at scale"]
N7["Practice note — priority queues"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX106"]
[Learning outcomes]
|
v
[36.1 Unstable queues]
|
v
[Lab 8 — queue growth]
|
v
[36.2 Little's law]
|
v
[36.3 Options]
|
v
[36.4 Protecting partners]
|
v
[36.5 Testing at scale]
|
v
[Practice note — priority queues]
|
v
[Worked example] --> [Exercise EX106]
Learning outcomes
You will (1) detect unstable queues where arrival exceeds service; (2) apply Little's law to estimate waiting time; (3) choose between scaling up, scaling out and shedding load; (4) protect partners when Lantern scales.
36.1 Unstable queues
If 120 objects per minute arrive at enrichment and only 100 per minute are served, the backlog grows by 20 each minute without limit. No amount of waiting fixes this; capacity or demand must change.
The lab prints exactly this pattern, and after five minutes the backlog is a hundred items and still rising. Kofi watches the trend rather than the size: a long queue that is shrinking is recovering, while a short one growing every minute is a problem. His alert fires when the backlog has grown for ten minutes in a row, before it becomes large enough for analysts to notice.
Lab 8 — queue growth
"""Lab 8 (ch 36): Little's law and backlog growth for a synthetic enrichment queue."""
arrival_per_min, service_per_min = 120, 100
backlog = 0
for minute in range(1, 6):
backlog += arrival_per_min - service_per_min
print(f"minute {minute}: backlog={backlog}")
L, lam = 400, 100 / 60 # items in system, throughput items/s after scaling fixes
print(f"Little's law wait W=L/lambda={L/lam:.0f} s at L={L}, lambda={lam:.2f}/s")
Run with python3 lab_queue.py. Verified output:
minute 1: backlog=20
minute 2: backlog=40
minute 3: backlog=60
minute 4: backlog=80
minute 5: backlog=100
Little's law wait W=L/lambda=240 s at L=400, lambda=1.67/s
36.2 Little's law
For a stable system, the average number in the system L equals throughput λ times average time in system W. With 400 items queued and 1.67 items per second served, average wait is about 240 seconds. The law needs no assumptions about arrival patterns, which makes it a robust check on dashboards.
Kofi uses it to check whether the dashboard's figures agree with each other. If the queue length, the throughput and the reported wait do not fit the law, one of the measurements is wrong.
36.3 Options
Scale up (faster hardware), scale out (more workers), or shed load (skip low-value enrichment, sample low-confidence indicators). Shedding by value is often cheapest: enrich high-confidence indicators first.
Kofi compares the cost of each option against the gain. A faster machine helps only if the work is limited by processing. More workers help only if the work can be split and the provider allows more queries. Shedding low-value work needs a clear rule about what counts as low value, agreed with analysts, and a record of what was skipped so it can be done later if needed.
36.4 Protecting partners
Scaling out collectors must not multiply load on partner TAXII servers. Lantern keeps one polling identity per partner and shares results internally, respecting partner rate limits.
Kofi's design separates collection from processing for this reason. However many internal workers Lantern runs, one collector per partner polls at the agreed rate and places objects on an internal queue. Partners see the same traffic whether BEN is quiet or in the middle of a campaign. If a partner raises its limits, the configuration changes after the agreement does, not before.
36.5 Testing at scale
Load tests use synthetic bundles generated to realistic sizes, run against the TAXII stub and internal components only.
Kofi's generator creates bundles with the same mix of object types and sizes that partners send, including a share of malformed objects, so the test exercises the reject path as well. Running against real partner servers would test their capacity rather than BEN's, and would breach the agreements. The results record the hardware, the code version and the generated volumes, so a later test can be compared fairly. Each run ends by checking that queues drained completely.
Practice note — priority queues
Two queues — high and low confidence — with workers serving high first keep important enrichment fast during bursts while low-value work waits or is skipped.
Capacity reviews
Every quarter Kofi compares peak arrival rates with service capacity and records the headroom percentage, so growth is planned before queues become unstable.
Worked example
During a large campaign, arrivals jump to 120 per minute. Kofi adds two workers (service 150 per minute) and switches low-confidence indicators to a deferred queue. The backlog drains in under ten minutes, and partner polling stays unchanged.
Synthetic case. Rates are invented.
Errors and limits
Little's law describes averages; tail latency needs percentiles. Scaling the consumer of a rate-limited external provider just produces more 429 responses.
Exercises
- EX106 — Backlog after five. What backlog does the lab show after five minutes, and why does it keep growing? Rubric: 3 = 100; arrival exceeds service by 20/min; 2 = number only; 1 = stable; 0 = none.
- EX107 — Little's law. Compute W for L = 300 items at λ = 2 items per second. Rubric: 3 = 150 s; 2 = method; 1 = 600; 0 = none.
- EX108 — Partner load. Why must scaling out collectors not add polling identities per partner? Rubric: 3 = would multiply partner load/breach rate limits; 2 = partial; 1 = fine; 0 = none.
Synthesis
Spot instability, use Little's law for waits, scale or shed by value, and keep partners' load constant.
Continue studying — chapter 36 of 50
← Chapter 35 (draft) · Course chapter index · Chapter 37 (draft) →