◇ OpenCyber Academy
Master's · Chapter 6 · English

Chapter 6 — Data collection and analysis

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 6 — Data collection and analysis

Lesson map

Lesson 06 published. Map, example, EX033.

flowchart TD
  N0["Learning outcomes"]
  N1["6.1 Collecting data"]
  N2["Lab — Stratified comparison"]
  N3["6.2 Stratified results"]
  N4["6.3 Other checks"]
  N5["6.4 Interpretation"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX033"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["6.1 Collecting data"]
  N2["Lab — Stratified comparison"]
  N3["6.2 Stratified results"]
  N4["6.3 Other checks"]
  N5["6.4 Interpretation"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX033"]

[Learning outcomes]
    |
    v
[6.1 Collecting data]
    |
    v
[Lab — Stratified comparison]
    |
    v
[6.2 Stratified results]
    |
    v
[6.3 Other checks]
    |
    v
[6.4 Interpretation]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX033]

Learning outcomes

You will (1) collect and code the main dataset; (2) analyse stratified results; (3) interpret with competing explanations in mind.

6.1 Collecting data

Tariq codes all forty protocols over six weeks, with a second coder double-coding a random quarter. Agreement on control type is complete, and on incidents one disagreement is resolved using the codebook example from the pilot. He records exposure time for three protocols that strengthened controls mid-window. Data and decisions are stored with the scripts. Disciplined collection gives the analysis a dataset whose every value can be traced to a source and a rule. Coders log time spent per protocol.

Lab — Stratified comparison

"""Stratify the control/incident comparison by protocol size (synthetic, read-only)."""
# (size, control, protocols, incidents)
ROWS = [
    ("small", "single key", 9, 4), ("small", "multisig", 6, 2), ("small", "multisig+timelock", 3, 1),
    ("large", "single key", 3, 1), ("large", "multisig", 10, 1), ("large", "multisig+timelock", 9, 0),
]

def main():
    for size in ("small", "large"):
        print(size)
        for s, c, n, i in ROWS:
            if s == size:
                print(f"  {c}: {i}/{n} ({100 * i / n:.1f}%)")
    small = sum(n for s, _, n, _ in ROWS if s == "small")
    print(f"share of single-key protocols that are small: {9}/{12}; small protocols overall: {small}/40")

if __name__ == "__main__":
    main()

Run with python3 lab_strata.py. Verified output:

small
  single key: 4/9 (44.4%)
  multisig: 2/6 (33.3%)
  multisig+timelock: 1/3 (33.3%)
large
  single key: 1/3 (33.3%)
  multisig: 1/10 (10.0%)
  multisig+timelock: 0/9 (0.0%)
share of single-key protocols that are small: 9/12; small protocols overall: 18/40

6.2 Stratified results

Stratifying by size changes the picture. Among small protocols, incidents occur in 4 of 9 single-key, 2 of 6 multi-signature and 1 of 3 timelocked protocols. Among large protocols, the rates are 1 of 3, 1 of 10 and 0 of 9. Nine of the twelve single-key protocols are small, while most timelocked ones are large. The association is weaker within strata than in the crude comparison, which shows that size explains part, though apparently not all, of the crude difference.

Among large protocols, one incident in three single-key protocols sets that rate at 33.3 percent, so even the stratified figures rest on very few events. Tariq therefore reports the stratified counts beside the crude ratio of 5.0.

6.3 Other checks

Restricting to protocols with at least one prior audit, the ordering of rates by control type persists but rests on very few incidents. Deployment year shows newer protocols more often start with timelocks, a pattern consistent with changing norms. For a subset, Lumen's alerts reveal two unreported small incidents, both in single-key protocols, suggesting reporting bias did not favour his hypothesis. Tariq presents these checks together so readers see how the evidence fits across tests. Each check is reported with its counts and limits.

6.4 Interpretation

Tariq concludes that stronger privileged controls are associated with fewer key-related incidents, that part of the crude association reflects size, and that the remaining association is uncertain because events are few. He recommends multi-signature with timelocks as consistent with EthTrust guidance and his data, while stating clearly that his study cannot prove causation. Interpretation that matches the strength of evidence earns more trust than a confident headline. He also notes that a larger dataset across several firms would be needed to estimate the effect precisely, and suggests how such data might be shared safely.

Worked example

Stratification shows weaker but persisting associations; nine of twelve single-key protocols are small; unreported incidents in single-key protocols argue against reporting bias favouring the hypothesis.

Synthetic case. Protocols and results are synthetic; EEA EthTrust is a real source.

Errors and limits

Presenting crude rates as the final result hides the role of protocol size.

Exercises

Synthesis

Stratified analysis and multiple checks give a nuanced, defensible interpretation of observational data.

Continue studying — chapter 06 of 08

← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →