Chapter 28 — Security operations and blue team: failure analysis
Lesson map
Lesson 28 published. Map, example, EX082.
flowchart TD
N0["Learning outcomes"]
N1["28.1 Post-incident review"]
N2["28.2 Timeline"]
N3["28.3 What failed"]
N4["28.4 Contributing factors"]
N5["28.5 Blameless approach"]
N6["28.6 Actions"]
N7["28.7 Follow-up"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX082"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["28.1 Post-incident review"]
N2["28.2 Timeline"]
N3["28.3 What failed"]
N4["28.4 Contributing factors"]
N5["28.5 Blameless approach"]
N6["28.6 Actions"]
N7["28.7 Follow-up"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX082"]
[Learning outcomes]
|
v
[28.1 Post-incident review]
|
v
[28.2 Timeline]
|
v
[28.3 What failed]
|
v
[28.4 Contributing factors]
|
v
[28.5 Blameless approach]
|
v
[28.6 Actions]
|
v
[28.7 Follow-up]
|
v
[Worked example] --> [Exercise EX082]
Learning outcomes
You will (1) analyse why detections or responses failed; (2) use timelines and contributing factors; (3) avoid blame in reviews; (4) turn findings into actions.
28.1 Post-incident review
After the campaign, Riverton holds a review. Its aim is learning: what happened, why defences and response worked or failed, and what to change. NIST SP 800-61r3 treats such lessons as part of continuous improvement.
Riverton's review was held a week after the campaign, when containment was complete but memories were still fresh. It included student analysts, the duty engineers, the service desk and the DPO. Its terms stated that the aim was to improve systems, not to assign fault, which shaped how openly people described their decisions that night.
28.2 Timeline
The review builds a timeline from email delivery (01:50) to last containment (s3310 at 11:20). Each entry cites its source.
Building it required matching times from the mail service, identity provider, SIEM, case system and service desk tickets, all converted to UTC. Lucas's case notes from the day supplied the analyst actions. Each entry cites its source so that anyone questioning a time can check it against the original export. The finished timeline showed the long quiet stretch between s3310's sign-in at 02:31 and any SOC action, which became the review's central question. A timeline built from memory would have blurred that gap.
28.3 What failed
s3310 was compromised at 02:31 but not contained until 11:20. Its attacker created a rule deleting messages containing "phishing" — hiding warnings — but no alert fired, because the correlation window had not yet existed and the z-score did not flag s3310 (it sent few messages).
The gap of almost nine hours for s3310 matters because its deletion rule was removing warnings, including a classmate's question, while the attacker could keep sending mail from the account. Detection had been built around the s2047 pattern of heavy sending, and an attacker who behaved more quietly fell through.
28.4 Contributing factors
No MFA for students; legacy protocol enabled; batch detection; no rule for keyword-deletion rules; only one analyst on the early shift.
None of these factors alone explains the outcome. Without MFA, stolen passwords worked; with legacy protocols, MFA could have been bypassed even if enrolled; batch detection delayed s2047's alert; the missing deletion-rule alert left s3310 unseen; and a single early analyst could not handle reports and alerts together. Each factor became an action. Listing them separately also shows which were already fixed.
28.5 Blameless approach
The review focuses on systems and decisions, not individuals. People speak openly when they are not blamed.
In the review, the analyst on the early shift explained that she had seen several phishing reports but had not linked them to compromised accounts, because the reports described the email rather than any account. Without blame, that insight was recorded: reports and account alerts lived in different queues, which made the connection hard to see. The action list later added a link between the two queues.
28.6 Actions
Each factor becomes an action with owner and date: MFA requirement, legacy shutdown, 15-minute checks, correlation rule, an extra early shift during campaigns.
The extra early shift during campaigns responds directly to the single-analyst factor. The fifteen-minute checks replace the morning batch that delayed s2047's alert. Each action names who will do it and by when, and the review report lists them in a table so their progress can be checked. Owners report progress at the monthly review, and an action whose date passes without completion is raised with Dr. Kim rather than silently extended.
28.7 Follow-up
Actions are tracked until closed and checked in the next exercise.
Dr. Kim reviews open actions monthly. The next purple exercise included a slow mailbox-rule attacker similar to s3310, to check that the new keyword-deletion alert fired. It did, and the action was closed with a reference to the exercise report. Actions closed without such a check are reopened. The MFA requirement was followed up differently: Lucas compared risky sign-in alerts before and after, as the controls chapter showed, and the falling count of foreign sign-ins without MFA was recorded as evidence that the action had worked.
Writing the review
The review report is short: timeline, what went well, what did not, contributing factors and actions. It is shared with IT, the service desk and the data protection officer, and a version without personal details is used in training.
Worked example
The review asks why s3310 was found at all. Answer: a classmate replied to s3310's phishing email asking if it was real; s3310 never saw the reply (deleted by the rule) but the classmate also reported it. The finding reinforces the value of reporting and of alerting on deletion rules.
Synthetic case. The review is invented.
Errors and limits
Reviews depend on available logs; unknowns are recorded as such.
Exercises
- EX082 — s3310 delay. Why was s3310 not contained until 11:20? Rubric: 3 = deletion rule hid warnings; no alert; low send volume missed by z-score; 2 = partial; 1 = unknown; 0 = none.
- EX083 — Contributing factors. List four contributing factors. Rubric: 3 = no MFA, legacy protocol, batch detection, no deletion-rule alert, one analyst (four); 2 = three; 1 = one; 0 = none.
- EX084 — How found. How was s3310 eventually found? Rubric: 3 = classmate's report; 2 = partial; 1 = tools; 0 = none.
Synthesis
Blameless failure analysis turns timelines and contributing factors into owned actions.
Continue studying — chapter 28 of 40
← Chapter 27 (draft) · Course chapter index · Chapter 29 (draft) →