Chapter 29 — Web3, blockchain and cryptoasset security: measurement

Chapter 29 — Web3, blockchain and cryptoasset security: measurement

Lesson map

Lesson 29 published. Map, example, EX085.

flowchart TD
  N0["Learning outcomes"]
  N1["29.1 Choosing metrics"]
  N2["29.2 Collecting reliably"]
  N3["29.3 Judging controls"]
  N4["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4
  N4 --> EX["Worked example"]
  EX --> Q["Exercise EX085"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["29.1 Choosing metrics"]
  N2["29.2 Collecting reliably"]
  N3["29.3 Judging controls"]
  N4["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX085"]

[Learning outcomes]
    |
    v
[29.1 Choosing metrics]
    |
    v
[29.2 Collecting reliably]
    |
    v
[29.3 Judging controls]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX085]

Learning outcomes

You will (1) define security and operational metrics; (2) collect them reliably; (3) use them to judge controls.

29.1 Choosing metrics

Jonas defines metrics that show whether controls work: percentage of privileged functions meeting the role policy; invariant violations detected per month; time from a queued privileged action to monitor confirmation; minting events blocked by the cap; reconciliation mismatches between database and ledger; and validator availability. Each metric has a definition, data source, collection method and target. He avoids counting activity, such as the number of transactions, as a security metric. The board approves the list, and Jonas explains each metric in one plain sentence in the monthly report.

29.2 Collecting reliably

Metrics are collected by scripts from ledger events, monitoring logs and configuration exports, run on a schedule with results stored and versioned. Jonas tests each collection script on synthetic data with known answers before using it. He records any change to a metric's definition, because a changed definition can create an apparent trend that is not real. Reliable collection is what makes metrics trustworthy enough to drive decisions by the board. Each script's output includes the data source, run date and definition version, so any figure can be traced back to how it was produced.

29.3 Judging controls

After three months of test operation, metrics show 100 percent policy conformance, zero invariant violations, a median monitor confirmation of nine hours and three minting batches blocked by the cap due to a meter clock fault. Jonas uses the last to show the cap working as intended and to prompt a fix to meter clocks. Metrics are valuable when they lead to action; he adds a short action column to each monthly report. The board reviews the report monthly and decides on actions, which Jonas tracks to completion.

Worked example

Six metrics are defined; three months of test operation show full policy conformance, no invariant violations and three cap blocks caused by a meter clock fault.

Synthetic case. Metrics and results are synthetic.

Errors and limits

Using transaction counts as a security metric says nothing about whether controls work.

Exercises

• EX085 — Metric example. Name one defined metric. Rubric: 3 = policy conformance percentage;; 2 = invariant violations;; 1 = transactions;; 0 = none.
• EX086 — Definition change. Why record metric definition changes? Rubric: 3 = changes can create false trends;; 2 = false trends;; 1 = cost;; 0 = none.
• EX087 — Cap blocks cause. What caused the three cap blocks? Rubric: 3 = a meter clock fault;; 2 = clock;; 1 = attack;; 0 = none.

Synthesis

Defined, reliably collected metrics show whether controls work and prompt improvements.

Continue studying — chapter 29 of 50

← Chapter 28 (draft) · Course chapter index · Chapter 30 (draft) →