Chapter 28 — Application and software supply chain security: failure analysis
Lesson map
Lesson 28 published. Map, example, EX082.
flowchart TD
N0["Learning outcomes"]
N1["28.1 Blameless analysis"]
N2["28.2 The token leak"]
N3["28.3 Root cause and factors"]
N4["28.4 Actions"]
N5["28.5 Near misses"]
N6["28.6 Sharing"]
N7["Tracking actions"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX082"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["28.1 Blameless analysis"]
N2["28.2 The token leak"]
N3["28.3 Root cause and factors"]
N4["28.4 Actions"]
N5["28.5 Near misses"]
N6["28.6 Sharing"]
N7["Tracking actions"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX082"]
[Learning outcomes]
|
v
[28.1 Blameless analysis]
|
v
[28.2 The token leak]
|
v
[28.3 Root cause and factors]
|
v
[28.4 Actions]
|
v
[28.5 Near misses]
|
v
[28.6 Sharing]
|
v
[Tracking actions]
|
v
[Worked example] --> [Exercise EX082]
Learning outcomes
You will (1) analyse incidents and near misses without blame; (2) find root causes and contributing factors; (3) design actions that prevent recurrence; (4) share lessons.
28.1 Blameless analysis
The goal is to understand how the system allowed a failure, not to blame a person. People make mistakes; systems should catch them.
Lena writes the analysis in neutral language: what happened, what made it possible and what would make it less likely. People involved are invited to explain their view without fear, because hidden mistakes cannot be fixed. The developer who pushed the token helps write the timeline. Lena uses a short template so every analysis follows the same structure and can be compared later.
28.2 The token leak
Timeline: test token added to config/test.env; .gitignore excluded the wrong path; commit pushed to public repository; server-side scanner flagged it two hours later; token revoked within fifteen minutes of the alert.
The timeline is built from logs, not memory: the commit time, the scanner alert and the provider's revocation record. Lena notes that the provider's logs showed no use of the token while it was exposed. Each entry in the timeline has its source noted, so anyone reading the analysis can check it.
28.3 Root cause and factors
Root cause: secrets were kept in files inside the repository folder. Contributing factors: no pre-commit hook; the .gitignore mistake; unclear guidance about test keys.
Each factor made the leak more likely, but none alone would have been enough. Lena separates them so that each can have its own action. The unclear guidance mattered because the developer believed a test key was harmless, which the analysis shows was a reasonable belief given what the society had written at the time. Fixing them separately helps.
28.4 Actions
Move all secrets to the secret store; add the pre-commit hook; write guidance; add a CI check that fails if any .env file is committed.
Each action has an owner and a date, and Lena checks at the next term's review that all were completed. The CI check is a second line behind the hook: even if a developer bypasses the hook, the build fails when an environment file appears in the repository. The guidance now explains why test keys still need protection, with the society's own incident as the example.
28.5 Near misses
The pay-client hash mismatch was a near miss handled well; its analysis records what worked (the check, not overriding it) so it is kept.
Recording what worked matters as much as recording failures, because a control that is never credited is easy to remove later as an inconvenience. Lena notes that a member could have overridden the check to get the build working, and that not doing so is the behaviour the society wants to encourage. Near misses are cheap lessons.
28.6 Sharing
A short write-up is shared with other student projects at the university.
The write-up removes names and details that could identify the developer and focuses on the lessons that apply to any student project: keep secrets out of the repository folder, scan before and after pushes and respect integrity checks. Grace circulates it through the security office, and two other student projects adopt the pre-commit hook within the term. Lena also offers to help those projects set up the scanner, and records the questions they ask for the next version of the write-up.
Tracking actions
Each action from the analysis has an owner and a date and is tracked in the society's issue tracker until closed. Lena checks progress at each fortnightly meeting. Three months later, all four actions are complete, and a test commit containing a fake .env file is correctly blocked by the CI check, confirming the action works.
Worked example
Lena uses the "five whys" method: why was the token public? It was committed. Why? .gitignore missed it. Why? The path was wrong. Why? Nobody tested the ignore rule. Why? There was no check for committed .env files. The last answer becomes the CI check.
Synthetic case. The analysis is synthetic.
Errors and limits
Root-cause methods can oversimplify; most incidents have several causes.
Exercises
- EX082 — Blameless goal. What is the goal of blameless analysis? Rubric: 3 = how system allowed failure, not blame; 2 = partial; 1 = vague; 0 = none.
- EX083 — Contributing factors. List the contributing factors to the leak. Rubric: 3 = no hook; .gitignore mistake; unclear guidance; 2 = two; 1 = one; 0 = none.
- EX084 — Five whys. What final action came from the five whys? Rubric: 3 = CI check for committed .env files; 2 = partial; 1 = vague; 0 = none.
Synthesis
Analyse failures without blame and turn causes into system changes.
Continue studying — chapter 28 of 40
← Chapter 27 (draft) · Course chapter index · Chapter 29 (draft) →