Chapter 32 — Threat intelligence and OSINT: reliability
Lesson map
Lesson 32 published. Map, example, EX094.
flowchart TD
N0["Learning outcomes"]
N1["32.1 Serial availability"]
N2["Lab 7 — availability"]
N3["32.2 Reading the lab"]
N4["32.3 Independence assumption"]
N5["32.4 SLOs and error budgets"]
N6["32.5 Graceful degradation"]
N7["Measuring what users feel"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX094"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["32.1 Serial availability"]
N2["Lab 7 — availability"]
N3["32.2 Reading the lab"]
N4["32.3 Independence assumption"]
N5["32.4 SLOs and error budgets"]
N6["32.5 Graceful degradation"]
N7["Measuring what users feel"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX094"]
[Learning outcomes]
|
v
[32.1 Serial availability]
|
v
[Lab 7 — availability]
|
v
[32.2 Reading the lab]
|
v
[32.3 Independence assumption]
|
v
[32.4 SLOs and error budgets]
|
v
[32.5 Graceful degradation]
|
v
[Measuring what users feel]
|
v
[Worked example] --> [Exercise EX094]
Learning outcomes
You will (1) compute availability of serial pipelines; (2) quantify the effect of redundancy; (3) set service-level objectives with error budgets; (4) decide where reliability investment pays off.
32.1 Serial availability
A request that must pass through every stage succeeds only if all stages are up, so availabilities multiply. Four stages that each look good can combine into something mediocre.
A stage that is up most of the time still fails occasionally, and the pipeline fails whenever any stage does. The lab puts numbers on that effect with four plausible figures. Kofi uses the calculation early in design to see which stage limits the whole, before spending money on stages that are already reliable.
Lab 7 — availability
"""Lab 7 (ch 32): availability of a synthetic CTI pipeline - serial stages and a redundant pair."""
stages = {"taxii-poll": 0.995, "parser": 0.999, "enrich": 0.98, "store": 0.999}
serial = 1.0
for a in stages.values():
serial *= a
print(f"serial availability={serial:.4f} (~{(1-serial)*8760:.0f} h/yr down)")
enrich_pair = 1 - (1 - stages["enrich"]) ** 2
improved = serial / stages["enrich"] * enrich_pair
print(f"with redundant enrich: {improved:.4f} (~{(1-improved)*8760:.0f} h/yr down)")
Run with python3 lab_avail.py. Verified output:
serial availability=0.9732 (~235 h/yr down)
with redundant enrich: 0.9926 (~65 h/yr down)
32.2 Reading the lab
The serial pipeline reaches 0.9732, roughly 235 hours of downtime a year — dominated by enrichment at 0.98. Duplicating enrichment (two independent instances, either can serve) lifts the pipeline to 0.9926, about 65 hours a year. Investing in the weakest stage gives the largest gain.
Kofi presented the result as hours rather than ratios, because managers found a figure of two hundred hours a year easier to judge than four decimal places. The same reasoning guides later spending: before buying more reliable storage, which is already close to perfect, the team asks which stage still contributes most of the lost hours.
32.3 Independence assumption
The redundancy formula assumes failures are independent. Two enrichment workers on the same host, or calling the same external provider, fail together. Real redundancy needs diverse hosts and, where possible, alternate providers.
Kofi checked the first redundancy design and found both enrichment instances calling the same external provider. If that provider went down, both would fail together, and the lab's improvement would vanish. His revised design adds a fallback that marks indicators as unenriched rather than waiting, which removes the shared dependency from the critical path. He states the independence assumption beside every availability figure he reports.
32.4 SLOs and error budgets
An SLO such as "99.5% of polls succeed each month" defines an error budget of 0.5%. When the budget is spent, feature work pauses and reliability work takes priority.
Kofi's team tracks the budget weekly on the same dashboard as the other metrics. When a partner outage used most of one month's budget, the team spent the next sprint adding retries and alerts instead of the planned reporting feature. The rule was agreed in advance with the CTI lead, so the pause did not need a fresh argument each time.
32.5 Graceful degradation
If enrichment fails, Lantern should still store and match indicators, marking them "unenriched", rather than stopping ingestion. Degraded service beats no service in a SOC.
Kofi's fallback was tested by stopping the enrichment service during a replay of synthetic traffic. Ingestion continued, matching continued and the indicators carried the unenriched flag. When enrichment returned, a background job filled in the missing context. Analysts reported that the flag was useful, because it told them to look up context by hand for urgent cases during the outage.
Measuring what users feel
Uptime of each process is not what analysts experience. Kofi adds a synthetic probe that injects a test indicator every fifteen minutes and checks it reaches the DNS policy. The probe's success rate is the availability users actually feel, and it catches failures between components that per-process health checks miss.
Worked example
Kofi proposes a second enrichment instance on a different host and a fallback mode that skips external enrichment when the provider is down. The measured monthly availability rises from 97.4% to 99.2% over the next quarter; these are measured figures, so they differ slightly from the 0.9732 and 0.9926 that Lab 7 predicts.
Synthetic case. Figures are illustrative.
Errors and limits
Availability numbers hide partial failures such as slow responses. Measure latency SLOs too.
Exercises
- EX094 — Serial result. Quote the lab's serial availability and approximate yearly downtime. Rubric: 3 = 0.9732, ~235 h; 2 = one; 1 = wrong; 0 = none.
- EX095 — Hidden dependency. Why might two enrichment workers not be independent? Rubric: 3 = shared host/provider/network → common-mode failure; 2 = vague; 1 = always independent; 0 = none.
- EX096 — Degraded mode. What should Lantern do when external enrichment is unavailable? Rubric: 3 = continue ingestion/matching; mark unenriched; retry later; 2 = continue; 1 = stop pipeline; 0 = none.
Synthesis
Multiply serial availabilities, fix the weakest stage with truly independent redundancy, manage error budgets and degrade gracefully.
Continue studying — chapter 32 of 50
← Chapter 31 (draft) · Course chapter index · Chapter 33 (draft) →