◇ OpenCyber Academy
Bachelor's · Chapter 28 · English

Chapter 28 — Failure analysis

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 28 — Failure analysis

Lesson map

Lesson 28 published. Map, example, EX082.

flowchart TD
  N0["Learning outcomes"]
  N1["28.1 Causes"]
  N2["28.2 Timeline"]
  N3["28.3 Blameless"]
  N4["28.4 Drift vs attack"]
  N5["28.5 Actions"]
  N6["28.6 Platform"]
  N7["28.7 Platform on-call in reviews"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX082"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["28.1 Causes"]
  N2["28.2 Timeline"]
  N3["28.3 Blameless"]
  N4["28.4 Drift vs attack"]
  N5["28.5 Actions"]
  N6["28.6 Platform"]
  N7["28.7 Platform on-call in reviews"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX082"]

[Learning outcomes]
    |
    v
[28.1 Causes]
    |
    v
[28.2 Timeline]
    |
    v
[28.3 Blameless]
    |
    v
[28.4 Drift vs attack]
    |
    v
[28.5 Actions]
    |
    v
[28.6 Platform]
    |
    v
[28.7 Platform on-call in reviews]
    |
    v
[Worked example] --> [Exercise EX082]

Learning outcomes

You will (1) classify tool/process/people causes; (2) run blameless reviews; (3) turn causes into fixes.

28.1 Causes

Tool misconfig, process gaps, unclear ROE, skill gaps — rarely "bad people".

Failure analysis at TRB starts by sorting causes into tool misconfiguration, process gaps, unclear rules of engagement and skill gaps. Personal blame is the last hypothesis, not the first. Sam has seen reviews waste an hour on who clicked what while a dropdown default kept inviting the same error. Naming the systemic bucket early keeps the room useful. When a cause really is individual negligence, the write-up still separates that from the platform and process fixes that would have made the mistake harder.

28.2 Timeline

Facts first.

A failure timeline lists observable facts with clocks before anyone offers motives. Logs, chat stamps and TrackScope events come first; interpretations wait. Sam refuses to let a review open with a story about who was careless. Once the facts are ordered, disagreements about cause become disagreements about evidence rather than about character. The timeline is pasted into the review notes so later readers can rebuild the argument. If a fact cannot be timed, it is marked uncertain instead of rounded into a neat narrative.

28.3 Blameless

Seek systemic fixes.

Blameless review means the primary output is a change to tools, defaults, checklists or training materials, not a performance note. Sam still coaches individuals, but the review ticket cannot close on coaching alone unless the template's justification box is filled. Teams that skip systemic fixes repeat the same incident with a new name attached. The practice is borrowed from operations culture and adapted to offensive engagements where shame travels fast. People tell the truth faster when the room is hunting broken systems.

28.4 Drift vs attack

Decide with evidence.

Some messy outcomes look like attacks when they are configuration drift, and the reverse is also true. The review asks what evidence would distinguish the two before it picks a label. Sam once chased an "intrusion" that was an overnight CMDB import of stale hosts. PulseMap and change tickets usually settle the question faster than intuition. Mislabeling drift as attack wastes SOC time; mislabeling attack as drift delays containment. The decision is written with the evidence that tipped it.

28.5 Actions

Owner + date per cause.

Every accepted cause leaves the review with a named owner and a due date. Actions without owners are wishes. Sam's template blocks "the team will improve" as an owner string; it wants a person or a queue. Dates are short enough to matter and long enough to be real. When an action slips, the slip is visible on the same board as engagement findings. Closure of the failure review waits until each action is done or explicitly deferred with a signature.

28.6 Platform

TrackScope bugs get engineering tickets too.

If the platform invited the mistake, engineering gets a ticket with the same seriousness as a finding. Sam files those tickets with the confusing control's screenshot and the review id. Platform on-call is invited to failure reviews even when the room thinks the error was human, because the UI often coached the click. Fixing TrackScope once prevents a generation of retraining. Operators learn to treat tooling defects as first-class outcomes of a review. A silent bugfix is still a bug fix.

28.7 Platform on-call in reviews

Failure reviews invite the platform on-call even when the mistake looks human. Often the UI invited the error.

Inviting on-call changes the conversation from "who misfired" to "what did the screen suggest". After one review, Sam's one-line copy change on a scope control removed a whole class of mis-clicks the following month. The on-call engineer would not have heard the pattern from a training ticket alone. Reviews that exclude platform staff tend to produce slide decks nobody ships. The invite is automatic on the review calendar template.

28.8 UI affordances

If three analysts make the same TrackScope mistake, UX changes before more training. Sam files UX tickets with screenshots of the confusing control.

Repeated identical mistakes are a design smell. Training the fourth person is slower and less reliable than changing the control. Sam's UX tickets always include a before screenshot and a proposed after state so designers are not guessing. The threshold of three is a heuristic, not magic; two can be enough when the blast radius is high. Affordances beat sermons when the clock is short and the UI is loud.

28.9 Five whys cap

Reviews stop at five whys or first actionable systemic fix. Endless why theatre is cut.

Five whys is a tool, not a ritual. Sam stops the chain when the room reaches a fix someone can own this sprint, even if that is why number three. Endless questioning becomes theatre that exhausts people without shipping change. The facilitator watches for the first actionable systemic answer and parks deeper philosophy for a separate design review. Caps keep failure analysis shorter than the incident it studies.

28.10 Stale inventory five whys

Five whys on the stale inventory stop at "CMDB API had no schema version field". Action: add schema version and a TrackScope checksum. Training the intern who imported the stale file is not the primary action.

The room could have blamed the intern who clicked import. Instead the fifth why landed on a missing schema version field and a checksum TrackScope never verified. Those two engineering actions became the review's primary output; the intern received a calm coaching note as a side item. Sam keeps this scene in the facilitator guide because it shows what blameless looks like when a junior is in the blast radius. Systems that cannot detect stale imports will keep hiring scapegoats.

28.11 Dropdown default to deny

UX ticket from failure review changed a dropdown default from "allow" to "deny". Mis-clicks dropped the following month. Defaults are destiny.

The control looked identical after the change; only the default option moved. Mis-clicks that had plagued scope selection fell sharply the next month without a new training module. Sam cites this when someone argues that UX work is cosmetic. Defaults silently authorise or block thousands of future clicks. Failure reviews that never touch defaults leave the same trap armed.

28.12 Before and after screenshots

UX tickets require a before/after screenshot of the control that confused people.

Words like "the dropdown is confusing" do not survive hand-off to a designer who was not in the room. Before and after screenshots make the request concrete and reviewable. Sam stores them under a ux- prefix in the same vault path as engagement evidence so access control stays familiar. Tickets lacking the pair bounce from triage. The discipline also forces the reporter to propose a clearer state instead of only complaining.

28.13 UX evidence path

Before/after UX screenshots live in the same vault path as engagement evidence with a ux- prefix.

A separate unofficial folder becomes a junk drawer nobody audits. Using the engagement vault with a clear prefix keeps retention, access reviews and legal holds consistent. Sam can point counsel at one tree when questions arise about what was captured during a messy week. The prefix also lets automation skip UX images when packing customer-facing evidence bundles. Small naming rules prevent large discovery headaches.

28.14 Platform-or-process checkbox

Failure review templates force a "platform or process change?" checkbox. If neither is checked, the review is incomplete. Training-only outcomes need explicit justification.

The checkbox is a speed bump against the lazy ending where everyone agrees to try harder. Leaving both boxes unchecked fails the template validation Sam wired into TrackScope. Training-only outcomes are allowed when justified in writing, for example a willful policy violation with no tooling gap. Most reviews should ship at least one platform or process change. The forced choice makes the culture visible in the data.

Worked example (synthetic TRB)

A false production hit trace shows a stale inventory feed; Sam files platform + CMDB actions.

Synthetic case — Riverstone Transit Bank (TRB). The bank, Sam Ito, TrackScope/FindingsDesk/PulseMap and all figures are invented for teaching.

Errors and limits

This book grants no authority to test real banks or payment systems. Labs use documentation-range addresses and synthetic data only; exploit code is never provided.

Exercises

Synthesis

Failure analysis produces owned systemic fixes, including to the platform.

Continue studying — chapter 28 of 40

← Chapter 27 (draft) · Course chapter index · Chapter 29 (draft) →