Chapter 28 — Threat intelligence and OSINT: failure analysis
Lesson map
Lesson 28 published. Map, example, EX082.
flowchart TD
N0["Learning outcomes"]
N1["28.1 Kinds of failure"]
N2["28.2 Cognitive biases"]
N3["28.3 Blameless review"]
N4["28.4 Timeline first"]
N5["28.5 Lessons that change things"]
N6["28.6 Near misses"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX082"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["28.1 Kinds of failure"]
N2["28.2 Cognitive biases"]
N3["28.3 Blameless review"]
N4["28.4 Timeline first"]
N5["28.5 Lessons that change things"]
N6["28.6 Near misses"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX082"]
[Learning outcomes]
|
v
[28.1 Kinds of failure]
|
v
[28.2 Cognitive biases]
|
v
[28.3 Blameless review]
|
v
[28.4 Timeline first]
|
v
[28.5 Lessons that change things]
|
v
[28.6 Near misses]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX082]
Learning outcomes
You will (1) analyse intelligence failures without blame; (2) distinguish collection, analysis and dissemination failures; (3) recognise cognitive biases; (4) produce actionable lessons.
28.1 Kinds of failure
Collection failure: the signal never arrived. Analysis failure: it arrived but was misjudged. Dissemination failure: the judgment was right but reached the wrong person or too late. Action failure: the right person received it but did not act. Each needs a different fix.
Amara uses these categories to sort the findings of every review, which helps the team see whether its problems cluster in one stage. In her first quarter, most findings were dissemination failures, such as advisories reaching a shared mailbox nobody owned, which pointed to process fixes rather than more analysis training. Each finding is tagged with its category in the review record.
28.2 Cognitive biases
Heuer's Psychology of Intelligence Analysis describes how analysts see what they expect, anchor on first impressions and favour evidence that confirms their view. Structured techniques such as analysis of competing hypotheses push analysts to test alternatives.
Amara's team uses competing hypotheses for any judgment that would lead to a major action, such as declaring an incident. Listing alternatives and the evidence for each makes it harder to fix on the first explanation. The technique takes time, so it is kept for decisions where an error would be costly. Peer review of drafts gives a lighter check for routine products.
28.3 Blameless review
Reviews ask what happened and why, not who to punish. People who fear blame hide mistakes, and the organisation stops learning.
At HCHT, reviews start by saying that the purpose is to improve the system. Amara saw the effect in her first review, where a colleague admitted missing an advisory because it arrived in an unmonitored mailbox; that admission led to the real fix. Without it, the review would have blamed one person and left the mailbox unmonitored, ready to cause the same failure again.
28.4 Timeline first
Build a timeline of when information arrived, who saw it and what they did. Failures often sit in gaps between teams.
Amara builds timelines from logs, tickets and email timestamps rather than memory. Each entry records what was known at that moment, not what was learned later, so the review judges decisions fairly. Gaps of several hours between one team receiving information and the next acting on it are marked, because they show where handovers need clearer owners. Amara uses a simple shared table.
28.5 Lessons that change things
"Be more careful" is not a lesson. "Add an owner field to every advisory and confirm receipt within one day" is.
Amara checks each lesson by asking who will do what by when, and how anyone will know it was done. Lessons that cannot answer those questions are rewritten. Each one gets an owner and a date, and the team reviews progress monthly. A lesson is closed only when the change has been made and checked, such as a test showing that advisories now get an owner within a day.
28.6 Near misses
A near miss — a phishing email noticed by chance before anyone clicked — is free learning. Reviewing near misses with the same structure as incidents reveals weak points before they cause harm, and staff find these reviews less threatening because nothing went wrong.
Amara reviewed one near miss in her internship. A nurse noticed an unusual sender and reported the message before anyone clicked. The review showed that the email filter had a rule for the sender's domain but it had been disabled during an outage. Restoring it, and adding email filter rules to the quarterly control check, came from that review, and the nurse's report was praised to encourage others.
Worked example
An HSSC advisory about a vulnerable remote-access appliance reached HCHT on a Friday. The general IT mailbox received it; nobody owned it. On Tuesday, the appliance was probed. The review finds a dissemination failure: no routing rule for appliance advisories. The lesson adds the appliance to the inventory with an owner and routes advisories by product match.
Synthetic case. The advisory and probing are invented.
Errors and limits
Hindsight makes signals look obvious. Judge decisions by what was known at the time.
Exercises
- EX082 — Failure type. Classify the Friday advisory failure and justify. Rubric: 3 = dissemination—right information, no owner/routing; 2 = type only; 1 = collection; 0 = none.
- EX083 — Bias example. Give an example of confirmation bias in CTI work. Rubric: 3 = specific example favouring confirming evidence about an actor/attribution; 2 = generic definition; 1 = unrelated; 0 = none.
- EX084 — Actionable lesson. Rewrite "be more careful with advisories" as an actionable lesson. Rubric: 3 = specific owner/process/timeframe; 2 = partial; 1 = still vague; 0 = none.
Synthesis
Classify failures by stage, guard against bias, review without blame and write lessons that change process.
Continue studying — chapter 28 of 40
← Chapter 27 (draft) · Course chapter index · Chapter 29 (draft) →