Chapter 4 — Web3, blockchain and cryptoasset security: competing explanations
Lesson map
Lesson 04 published. Map, example, EX021.
flowchart TD
N0["Learning outcomes"]
N1["4.1 The crude comparison"]
N2["Lab — Crude incident rates"]
N3["4.2 Reading the crude result"]
N4["4.3 Competing explanations"]
N5["4.4 Testing alternatives"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX021"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["4.1 The crude comparison"]
N2["Lab — Crude incident rates"]
N3["4.2 Reading the crude result"]
N4["4.3 Competing explanations"]
N5["4.4 Testing alternatives"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX021"]
[Learning outcomes]
|
v
[4.1 The crude comparison]
|
v
[Lab — Crude incident rates]
|
v
[4.2 Reading the crude result]
|
v
[4.3 Competing explanations]
|
v
[4.4 Testing alternatives]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX021]
Learning outcomes
You will (1) describe the crude association; (2) identify confounders; (3) plan analyses that test alternatives.
4.1 The crude comparison
Tariq first summarises incidents by control type without adjustment. A standard-library script reads the synthetic counts and prints rates and a risk ratio. Crude comparisons are a starting point, showing whether there is any association to explain, but they mix the effect of the control with everything else that differs between groups. He treats the result as a question rather than an answer. Tariq keeps the output with its date and the version of the data file. He also writes down, before running the stratified analysis, what pattern each competing explanation would predict, so he cannot adjust predictions after seeing the results.
Lab — Crude incident rates
"""Admin-control type versus key-related incidents in 40 synthetic protocols (read-only)."""
# control type: (protocols, with key-related incident)
DATA = {"single key": (12, 5), "multisig": (16, 3), "multisig+timelock": (12, 1)}
def main():
for k, (n, i) in DATA.items():
print(f"{k}: {i}/{n} incidents ({100 * i / n:.1f}%)")
n = sum(v[0] for v in DATA.values()); i = sum(v[1] for v in DATA.values())
print(f"all: {i}/{n} ({100 * i / n:.1f}%)")
r1 = DATA["single key"][1] / DATA["single key"][0]
r3 = DATA["multisig+timelock"][1] / DATA["multisig+timelock"][0]
print(f"risk ratio single key vs multisig+timelock: {r1 / r3:.1f}")
if __name__ == "__main__":
main()
Run with python3 lab_controls.py. Verified output:
single key: 5/12 incidents (41.7%)
multisig: 3/16 incidents (18.8%)
multisig+timelock: 1/12 incidents (8.3%)
all: 9/40 (22.5%)
risk ratio single key vs multisig+timelock: 5.0
4.2 Reading the crude result
The run shows key-related incidents in 5 of 12 single-key protocols (41.7 percent), 3 of 16 multi-signature protocols (18.8 percent) and 1 of 12 with multi-signature and timelock (8.3 percent), 9 of 40 overall. The crude risk ratio for single key against multi-signature with timelock is 5.0. Tariq notes how few events lie behind these percentages: one incident more or less in the timelock group would change the ratio dramatically, so the estimate is fragile. He reports the counts alongside every percentage.
With a single timelocked incident, the ratio of 5.0 rests on one event: a second incident in that group would halve it to 2.5.
4.3 Competing explanations
Tariq lists explanations other than the control itself. Size: larger protocols may have bigger teams and more audits, and also adopt stronger controls. Care: teams that adopt timelocks may be generally more careful. Reporting: larger protocols may attract more scrutiny, so their incidents are more likely to be reported, which would work against his finding. Change over time: newer protocols may start with stronger controls. Each explanation predicts a different pattern, which tells him what to check. He writes each in one sentence.
4.4 Testing alternatives
For size, he will stratify by value managed. For care, he will use audit count before the window as a proxy and check whether the association persists among audited protocols. For reporting, he will compare reported incidents with Lumen's monitoring alerts for a subset. For change over time, he will check deployment year. Matching each explanation to a test makes the analysis plan concrete and shows examiners that alternatives were taken seriously, not just mentioned. Dr Larsen approves the plan.
Worked example
Crude rates fall from 41.7% for single key to 8.3% for multisig with timelock (risk ratio 5.0), and four competing explanations are each matched to a planned test.
Synthetic case. Protocols and counts are synthetic.
Errors and limits
Reporting a crude risk ratio as the effect of a control ignores confounding by size and care.
Exercises
- EX021 — Crude rates. What were the crude incident rates? Rubric: 3 = 41.7%, 18.8%, 8.3%;; 2 = falling;; 1 = equal;; 0 = none.
- EX022 — Risk ratio. What crude risk ratio did the lab report? Rubric: 3 = 5.0;; 2 = five;; 1 = 1.0;; 0 = none.
- EX023 — Fragility. Why is the estimate fragile? Rubric: 3 = few events, one incident would change it a lot;; 2 = small counts;; 1 = vague;; 0 = none.
- EX024 — Reporting direction. How might reporting bias work here? Rubric: 3 = against the finding, larger protocols report more;; 2 = against;; 1 = for;; 0 = none.
- EX025 — Care proxy. What proxy measures care? Rubric: 3 = audit count before the window;; 2 = audits;; 1 = age;; 0 = none.
- EX026 — Size test. How is the size explanation tested? Rubric: 3 = stratify by value managed;; 2 = stratify;; 1 = ignore;; 0 = none.
Synthesis
Crude associations raise questions; competing explanations matched to tests turn them into an analysis plan.
Continue studying — chapter 04 of 08
← Chapter 03 (draft) · Course chapter index · Chapter 05 (draft) →