Chapter 28 — Web3, blockchain and cryptoasset security: failure analysis
Lesson map
Lesson 28 published. Map, example, EX082.
flowchart TD
N0["Learning outcomes"]
N1["28.1 A synthetic failure"]
N2["28.2 Root and contributing causes"]
N3["28.3 Turning findings into improvements"]
N4["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4
N4 --> EX["Worked example"]
EX --> Q["Exercise EX082"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["28.1 A synthetic failure"]
N2["28.2 Root and contributing causes"]
N3["28.3 Turning findings into improvements"]
N4["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX082"]
[Learning outcomes]
|
v
[28.1 A synthetic failure]
|
v
[28.2 Root and contributing causes]
|
v
[28.3 Turning findings into improvements]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX082]
Learning outcomes
You will (1) analyse a failure without blame; (2) identify root and contributing causes; (3) turn findings into improvements.
28.1 A synthetic failure
In a synthetic exercise, the issuing service sends a mint batch containing one week's shifts twice, because a scheduled job restarted after a server update. Keyholders sign without noticing, since the batch looked normal. The contract's weekly cap blocks the second half, so only part of the duplicate is minted. Daily reconciliation detects 310 extra vouchers the next morning. The board pauses transfers for two hours, the extra vouchers are removed with the members' agreement, and the system resumes. No member loses vouchers. Sofia leads the analysis of what happened and why.
28.2 Root and contributing causes
Sofia uses a blameless approach: the goal is to understand the system, not to punish people. The root cause is the issuing service lacking a check that a week has already been processed. Contributing causes include the restart after an update, keyholders having no summary showing whether a week was new, and reconciliation running only daily. She also notes what worked: the weekly cap limited the error and reconciliation detected it. Recording what worked matters, because those controls should be kept and strengthened rather than removed in future changes to reduce effort.
28.3 Turning findings into improvements
Sofia proposes three improvements: the issuing service records processed weeks and refuses repeats; each batch sent to keyholders includes a summary showing the week number and whether it was already processed; and reconciliation runs every hour. Each improvement has an owner and a deadline. She writes a short report in plain language for members, explaining what happened, that no one lost vouchers and what is changing. Clear communication after a failure maintains trust. The improvements are later tested on the test network with a simulated restart, confirming the repeat is refused.
Worked example
A duplicated batch adds 310 extra vouchers, limited by the cap and caught by reconciliation; fixes add repeat refusal, batch summaries and hourly checks.
Synthetic case. The failure and response are synthetic.
Errors and limits
Blaming individuals for a system gap hides the root cause and discourages honest reporting.
Exercises
- EX082 — Extra vouchers. How many extra vouchers were detected? Rubric: 3 = 310;; 2 = three hundred ten;; 1 = 3,100;; 0 = none.
- EX083 — Root cause. What was the root cause? Rubric: 3 = no check that a week was already processed;; 2 = missing repeat check;; 1 = a keyholder;; 0 = none.
- EX084 — Controls that worked. Which controls limited and detected the error? Rubric: 3 = the weekly cap and reconciliation;; 2 = cap;; 1 = vague;; 0 = none.
Synthesis
Blameless analysis finds causes, keeps what worked and drives tested improvements.
Continue studying — chapter 28 of 40
← Chapter 27 (draft) · Course chapter index · Chapter 29 (draft) →