Chapter 28 — Analysis of missed signals and response failures
Lesson map
Lesson 28 published. Map, example, EX082.
flowchart TD
N0["Learning outcomes and connection"]
N1["28.1 Hindsight bias"]
N2["28.2 Local rationality"]
N3["28.3 Coverage and thresholds"]
N4["28.4 Corrective actions"]
N5["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX082"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["28.1 Hindsight bias"]
N2["28.2 Local rationality"]
N3["28.3 Coverage and thresholds"]
N4["28.4 Corrective actions"]
N5["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX082"]
[Learning outcomes and connection]
|
v
[28.1 Hindsight bias]
|
v
[28.2 Local rationality]
|
v
[28.3 Coverage and thresholds]
|
v
[28.4 Corrective actions]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX082]
Learning outcomes and connection
You will be able to (1) recognize hindsight bias and reconstruct what was knowable at each decision point; (2) analyse detection coverage and thresholds; (3) include human factors such as workload and ownership; and (4) propose measurable corrective actions. Chapter 20 compared cases; this chapter turns the method inward on one's own failures. Chapter 29 builds the operational review cycle that keeps corrective actions alive.
28.1 Hindsight bias
After an outcome is known, people judge it as more predictable than it was. Fischhoff's experiments (1975) showed that knowing an outcome raises its perceived likelihood and leads observers to overestimate what decision-makers should have foreseen. In post-incident reviews this produces the familiar verdict "the alert was obvious; the analyst should have acted". The corrective is to reconstruct, for each decision point, what the person knew, what competed for their attention and what rules they were following.
In the FS-2 review, the reconstruction keeps the 2 April ransomware out of the 23:40 picture.
28.2 Local rationality
Safety science (for example Dekker's work on "just culture" and the "new view" of human error) holds that people's actions made sense to them at the time given their goals, knowledge and focus. The question becomes: what made this action reasonable then? The answer usually points to system conditions — alert volume, unclear ownership, missing context — that can be changed.
Applied to the FS-2 alert, the question is why closing it as probable IT activity made sense at 23:52 on 28 March. The answer lies in the rule then in force, an empty owner field, no list of approved tools and 64 alerts for one analyst. Each of these can be changed; the analyst's character cannot usefully be.
28.3 Coverage and thresholds
A signal can be missed because no detection existed (coverage gap), because a detection existed but its threshold was not met, because it fired but was lost in volume, or because it reached someone without authority or context. Each has a different fix.
The FS-2 miss was not a coverage gap: a detection existed and fired. It reached an analyst who lacked the ownership information and was bound by a closure rule that did not require confirmation, while the queue held 64 alerts. Classifying it this way points to fixes in context and rules rather than to writing new detections.
28.4 Corrective actions
Good corrective actions are specific, owned, dated and measurable: "add owner field to asset inventory for all servers; owner populated for 100% of production servers by 30 June; verified by quarterly report". Avoid actions such as "be more vigilant" or "retrain staff" without a defined change.
The FS-2 review shows the form: owners populated for all production servers by 30 June with a quarterly check, an approved list of remote-access tools, a closure rule requiring confirmation for server software installation, and a review of night-shift volume. Each names something that will be different and how anyone can tell.
28.5 Worked example (synthetic): an ignored alert
Fictional. On 2 April, Harbourline suffered ransomware on its file server. Review findings: on 28 March an alert "new remote-access tool installed on FS-2" fired at 23:40 and was closed at 23:52 as "probably IT activity" by the night analyst. FS-2's owner field in the inventory was blank.
Decision-point reconstruction (28 March 23:40): - Knowable: alert text; FS-2's role (file server); no change ticket for remote-access tools that night (the change system was searchable). - Not easily knowable: who owned FS-2 (blank field); whether IT staff used that tool (no list of approved tools). - Competing demands: 64 alerts in the queue; one analyst on shift. - Rule in force: "close informational alerts that appear to be administrative activity" — no requirement to confirm with the owner.
Conclusion: the analyst followed the rule with the information available. System factors: missing ownership, no approved-tool list, a closure rule that did not require confirmation for server software installation, and workload.
Corrective actions: (1) owner populated for all production servers (target 100% by 30 June, quarterly check); (2) approved remote-access tool list, with alerts on others raised to medium severity; (3) closure of "administrative activity" on servers requires owner or change-ticket confirmation, with a next-business-day follow-up allowed at night; (4) review night-shift alert volume against staffing.
28.6 Common errors and limits
- Blaming individuals for system conditions.
- Counterfactual reasoning that assumes knowledge available only after the outcome.
- Corrective actions without owners or measures.
-
The opposite error: refusing to discuss individual performance at all; just culture distinguishes honest error from reckless or deliberate violations.
-
Recording the outcome, such as the 2 April ransomware, in the reconstruction of what the analyst could know on 28 March.
- Accepting a corrective action because it sounds serious, without checking that it changes the conditions found.
- Closing the review when actions are written, rather than when they are verified.
Exercises — chapter 28
Attempt each exercise before opening the worked results. Every case, organization, person, address and number in an exercise is synthetic unless a cited public source is named.
EX082 — Reconstruct a decision point. Inputs: §28.5, decision to close the alert. Steps: fill a table of knowable, not knowable, competing demands and rules for the analyst. Rubric: 3 = matches available facts, no hindsight information; 2 = one hindsight item; 1 = blame narrative; 0 = none.
EX083 — Classify the miss. Steps: classify the 28 March miss as coverage gap, threshold, volume or context/authority failure, with reasons. Rubric: 3 = primarily context/ownership and closure-rule failure, with volume contributing; not a coverage gap; 2 = one category without nuance; 1 = "coverage gap"; 0 = none.
EX084 — Measurable actions. Steps: rewrite "the SOC must be more careful with remote-access alerts" as two measurable corrective actions. Rubric: 3 = specific, owned, dated, measured; 2 = missing owner or measure; 1 = still vague; 0 = none.
Chapter synthesis
Review failures from the decision-maker's point of view at the time, classify the miss, fix the system conditions that made the error reasonable, and write corrective actions that are owned, dated and measured.
Continue studying — chapter 28 of 40
← Chapter 27 · Course chapter index · Chapter 29 →
Continue to the next authored lesson.