Chapter 47 — Failure and recovery trial
Lesson map
Lesson 47 published. Map, example, EX139.
flowchart TD
N0["Learning outcomes"]
N1["47.1 The trial"]
N2["47.2 During the failure"]
N3["47.3 Recovery"]
N4["47.4 Improvements"]
N0 --> N1 --> N2 --> N3 --> N4
N4 --> EX["Worked example"]
EX --> Q["Exercise EX139"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["47.1 The trial"]
N2["47.2 During the failure"]
N3["47.3 Recovery"]
N4["47.4 Improvements"]
N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX139"]
[Learning outcomes]
|
v
[47.1 The trial]
|
v
[47.2 During the failure]
|
v
[47.3 Recovery]
|
v
[47.4 Improvements]
|
v
[Worked example] --> [Exercise EX139]
Learning outcomes
You will (1) run a failure and recovery trial on governed data; (2) measure recovery; (3) improve recovery procedures.
47.1 The trial
Kofi runs a trial in which the pressure data platform fails during a simulated high-demand day. The synthetic scenario removes the main historian for three hours. The trial tests whether operators can continue safely using degraded mode procedures, whether data are recovered completely afterwards and whether governance records show what happened. Participants include two operators, a data steward and the on-call engineer. Prof. Varga observes and records timings without intervening, so the trial reflects how the team really responds. Everyone is briefed beforehand.
47.2 During the failure
Operators switch to local displays and manual readings within fifteen minutes, following the procedure. The on-call engineer receives the alert promptly but spends time finding the recovery instructions, which were stored on the failed server. The steward marks affected data in the log. Kofi notes that keeping recovery instructions on the system they recover is a classic mistake, and that the trial revealed it in a safe setting rather than during a real incident. He records the delay in finding the instructions with its timing.
47.3 Recovery
After the historian returns, buffered readings from site devices are reloaded. Kofi reconciles the reloaded readings with site buffers and finds a short gap where one buffer had filled and overwritten older data. He measures recovery time and completeness and records the gap with its cause. Network operations agrees to increase buffer size at critical sites. He notes that recovery is not complete until data are reconciled, not merely when systems restart. The reconciliation results are attached to the trial report for the governance board.
47.4 Improvements
Kofi records improvements: recovery instructions moved to a separate location with printed copies, larger buffers at critical sites and a reconciliation step added to the recovery runbook. A second short trial confirms that the engineer finds instructions quickly. Prof. Varga notes that trials are valuable because they expose dependencies that documents conceal, and that governance of data includes planning for the data's survival through failures, not only its quality in normal operation. The board asks that a similar trial be run every year with different participants and scenarios.
Worked example
Kofi runs a historian failure trial, finds recovery instructions stored on the failed server and a buffer overwrite gap, and improves storage of instructions, buffer sizes and recovery reconciliation.
Synthetic case. The trial is synthetic.
Errors and limits
Declaring recovery when systems restart ignores lost data; reconcile first.
Exercises
- EX139 — Instruction problem. Where were recovery instructions stored? Rubric: 3 = on the failed server;; 2 = partly correct;; 1 = vague;; 0 = none.
- EX140 — Data gap cause. What caused the data gap? Rubric: 3 = a buffer filled and overwrote older data;; 2 = partly correct;; 1 = vague;; 0 = none.
- EX141 — Recovery definition. When is recovery complete? Rubric: 3 = when data are reconciled;; 2 = partly correct;; 1 = restart;; 0 = none.
Continue studying — lesson 47 (chapter 47 of 50)
← Chapter 46 · Course chapter index · Chapter 48 →
Continue to the next authored lesson.