Chapter 37 — Web3, blockchain and cryptoasset security: operations

Chapter 37 — Web3, blockchain and cryptoasset security: operations

Lesson map

Lesson 37 published. Map, example, EX109.

flowchart TD
  N0["Learning outcomes"]
  N1["37.1 Watching privileged actions"]
  N2["Lab — Queued action monitor"]
  N3["37.2 Triage"]
  N4["37.3 Routine checks"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX109"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["37.1 Watching privileged actions"]
  N2["Lab — Queued action monitor"]
  N3["37.2 Triage"]
  N4["37.3 Routine checks"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX109"]

[Learning outcomes]
    |
    v
[37.1 Watching privileged actions]
    |
    v
[Lab — Queued action monitor]
    |
    v
[37.2 Triage]
    |
    v
[37.3 Routine checks]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX109]

Learning outcomes

You will (1) monitor privileged actions against change records; (2) triage alerts; (3) run routine operational checks.

37.1 Watching privileged actions

In operation, the most important events to watch are queued privileged actions, because the timelock gives a window to stop or respond to unwanted changes. Jonas writes a standard-library script that compares queued actions with the approved change calendar and flags any that lack a matching approved change reference. It reads only synthetic queue and calendar data and prints alerts for staff, who then check with the board. Jonas tests the script on a small queue with known matches and mismatches first, and schedules it to run every hour so that the timelock window leaves plenty of time to react.

Lab — Queued action monitor

"""Flag queued privileged actions that are not on the approved change calendar (synthetic, read-only)."""
APPROVED = {("Credit", "upgrade", "CHG-31"), ("Vault", "withdrawReserve", "CHG-33")}
QUEUED = [("Credit", "upgrade", "CHG-31", 49), ("Market", "upgrade", None, 25),
          ("Vault", "withdrawReserve", "CHG-33", 72), ("Credit", "mint", "CHG-34", 48)]

def main():
    for c, f, chg, hours in QUEUED:
        ok = (c, f, chg) in APPROVED
        print(f"{c}.{f} ref={chg} eta={hours}h -> {'expected' if ok else 'ALERT'}")
    print(f"alerts {sum((c, f, g) not in APPROVED for c, f, g, _ in QUEUED)} of {len(QUEUED)}")

if __name__ == "__main__":
    main()

Run with python3 lab_monitor.py. Verified output:

Credit.upgrade ref=CHG-31 eta=49h -> expected
Market.upgrade ref=None eta=25h -> ALERT
Vault.withdrawReserve ref=CHG-33 eta=72h -> expected
Credit.mint ref=CHG-34 eta=48h -> ALERT
alerts 2 of 4

37.2 Triage

The run checks four queued actions and raises two alerts. Market.upgrade has no change reference at all, and Credit.mint carries CHG-34, which is not on the approved calendar. Jonas's triage procedure contacts the keyholders who approved each action. The Market upgrade turns out to be a test-network action mistakenly queued on production by a developer; it is cancelled within the timelock. CHG-34 was approved by the board the day before but not yet added to the calendar, which is then updated.

37.3 Routine checks

Jonas writes an operations runbook with daily, weekly and monthly checks: daily review of alerts and node health; weekly reconciliation and invariant replay; monthly role policy check, metrics report and key holder availability confirmation; and quarterly signing drills. Each check has an owner and records its result. Routine checks keep controls working between incidents, and the records provide evidence that the system is operated as designed. Jonas reviews the records each quarter with the board to confirm checks are happening, and adjusts the runbook when a check proves unhelpful or a new risk appears, recording each change.

Worked example

The monitor flags two of four queued actions; one mistaken upgrade is cancelled within the timelock and one calendar gap is corrected.

Synthetic case. Actions, calendar and runbooks are synthetic.

Errors and limits

Ignoring queued privileged actions wastes the window a timelock provides.

Exercises

• EX109 — Alert count. How many alerts did the monitor raise? Rubric: 3 = two of four;; 2 = two;; 1 = vague;; 0 = none.
• EX110 — Market upgrade outcome. What happened with the Market upgrade? Rubric: 3 = mistaken test action, cancelled within timelock;; 2 = cancelled;; 1 = executed;; 0 = none.
• EX111 — Weekly checks. What happens weekly? Rubric: 3 = reconciliation and invariant replay;; 2 = reconciliation;; 1 = drills;; 0 = none.

Synthesis

Monitoring queued actions against approvals, careful triage and routine checks keep operations safe.

Continue studying — chapter 37 of 50

← Chapter 36 (draft) · Course chapter index · Chapter 38 (draft) →