Chapter 13 — Web3, blockchain and cryptoasset security: troubleshooting

Lesson map

Lesson 13 published. Map, example, EX049.

flowchart TD
  N0["Learning outcomes"]
  N1["13.1 A reported problem"]
  N2["13.2 Testing hypotheses"]
  N3["13.3 Recording and solving"]
  N4["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4
  N4 --> EX["Worked example"]
  EX --> Q["Exercise EX049"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["13.1 A reported problem"]
  N2["13.2 Testing hypotheses"]
  N3["13.3 Recording and solving"]
  N4["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX049"]

[Learning outcomes]
    |
    v
[13.1 A reported problem]
    |
    v
[13.2 Testing hypotheses]
    |
    v
[13.3 Recording and solving]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX049]

Learning outcomes

You will (1) troubleshoot methodically; (2) use telemetry to test hypotheses; (3) record solutions.

13.1 A reported problem

Counter staff report that returned tools sometimes take more than ten minutes to show a released deposit, while it usually takes under one. Aiko defines the symptom precisely: delays occur only for some returns, mostly in the afternoon. She gathers telemetry for the affected period: block production, node heights, deposit service logs and database timings. A precise symptom narrows the search. Mr Kowal reminds her to change nothing until she has data, because changes made in a hurry can hide the real cause and create new problems.

13.2 Testing hypotheses

Aiko lists hypotheses: a slow node, a stalled deposit service queue, database locking or slow signer approvals for releases above the threshold. Telemetry shows nodes healthy and blocks regular. Deposit service logs show releases above the threshold waiting for staff multisig approval, while small releases are quick. Afternoon delays match times when only one signer is on duty. The cause is process, not technology: large releases need two signers and only one is present. Testing each hypothesis against data avoids blaming the ledger for a staffing pattern.

13.3 Recording and solving

The manager adjusts rosters so two signers are available during opening hours, and the threshold for multisig release is reviewed and kept, since it protects larger deposits. Aiko records the symptom, hypotheses, evidence, cause and solution in the troubleshooting log, with a summary line for quick searching. Delays drop to under a minute for all but a few cases. She notes that many technical-looking problems have human or process causes, and good troubleshooting follows the evidence wherever it leads, even away from the technology.

Worked example

Delayed releases are traced to large releases waiting for a second signer in the afternoon, solved by rostering two signers during opening hours.

Synthetic case. The problem and solution are synthetic.

Errors and limits

Assuming a technical cause can hide a process problem such as signer availability.

Exercises

• EX049 — Symptom definition. How does Aiko define the symptom?
  Show marking criteria

  Rubric: 3 = some returns over ten minutes, mostly afternoons;; 2 = afternoon delays;; 1 = website slow;; 0 = none.
• EX050 — Hypothesis list. Name one hypothesis tested.
  Show marking criteria

  Rubric: 3 = stalled deposit service queue;; 2 = slow node;; 1 = sunspots;; 0 = none.
• EX051 — Actual cause. What was the actual cause?
  Show marking criteria

  Rubric: 3 = only one signer on duty for large releases;; 2 = signer availability;; 1 = slow node;; 0 = none.
• EX052 — Threshold decision. What happened to the multisig threshold?
  Show marking criteria

  Rubric: 3 = reviewed and kept;; 2 = kept;; 1 = removed;; 0 = none.

Synthesis

Methodical troubleshooting follows evidence to the real cause, technical or not.

Continue studying — chapter 13 of 18

← Chapter 12 (draft) · Course chapter index · Chapter 14 (draft) →