Chapter 47 — Failure and recovery trial | Engineering Identity and access management

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 47 published. Map, example, EX139.

flowchart TD
  N0["Learning outcomes"]
  N1["47.1 Trial design"]
  N2["47.2 Running the trial"]
  N3["47.3 Measuring against targets"]
  N4["47.4 Lessons"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX139"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["47.1 Trial design"]
  N2["47.2 Running the trial"]
  N3["47.3 Measuring against targets"]
  N4["47.4 Lessons"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX139"]

[Learning outcomes]
    |
    v
[47.1 Trial design]
    |
    v
[47.2 Running the trial]
    |
    v
[47.3 Measuring against targets]
    |
    v
[47.4 Lessons]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX139]

Learning outcomes

You will (1) run a controlled failure and recovery trial; (2) measure recovery against targets; (3) record lessons.

47.1 Trial design

Before the project ends, Sana runs a controlled trial of failure and recovery in the test tenant, mirroring production configuration. Scenarios include loss of the HR feed for a day, a connector outage during a busy leaver period, an expired signing certificate and an unreachable provider requiring emergency access. Each scenario has a trigger, expected behaviour, recovery steps and target times. The council's identity team runs the recovery while Sana observes, because the aim is to test whether the team, not just the technology, can recover.

47.2 Running the trial

The trial runs over two days. The HR feed loss raises an alert within forty minutes, and no staff are wrongly disabled. The connector outage queues twelve synthetic leaver operations, which complete within thirty minutes of restoration. The expired certificate stops sign-in in the test tenant; renewal and publication take fifty-five minutes against a target of one hour. Emergency access works, but the team takes twenty minutes to locate the correct runbook page, longer than expected under pressure. All timings are logged for later analysis.

47.3 Measuring against targets

Sana tabulates each scenario against its target time and expected behaviour. Three of four meet targets; the certificate renewal meets it narrowly. The emergency access scenario meets the functional goal but reveals a usability problem with the runbook. She reports these as measurements, not impressions, and links each to the requirement it tests. Narrow margins are treated as warnings: a renewal that takes fifty-five minutes in a calm trial may take longer during a real incident at night. Results are shared with the council.

47.4 Lessons

Lessons become actions: the runbook gets a one-page emergency card at the front; certificate renewal is automated further to shorten it; and the trial is scheduled annually. Sana also notes a positive lesson: the failure-safe design for the HR feed worked exactly as intended, justifying the effort spent designing it. Lessons are recorded in the decision log and the operations guide. Trials like this build confidence that the system and team can handle real failures, which is the purpose of resilience engineering.

Worked example

Three of four failure scenarios meet targets; certificate renewal takes 55 of 60 minutes and emergency access takes twenty minutes to find the runbook, leading to automation and a one-page emergency card.

Synthetic case. Scenarios, times and outcomes are synthetic.

Errors and limits

Testing only the technology and not the team's ability to recover overstates resilience.

Exercises

EX139 — Feed loss outcome. What happened in the HR feed loss scenario?
Show marking criteria
Rubric: 3 = alert within forty minutes, nobody wrongly disabled;; 2 = safe failure;; 1 = mass disable;; 0 = none.
EX140 — Certificate timing. How long did certificate renewal take against its target?
Show marking criteria
Rubric: 3 = 55 minutes vs 60;; 2 = under target;; 1 = over;; 0 = none.
EX141 — Runbook fix. What fix addresses the slow runbook lookup?
Show marking criteria
Rubric: 3 = one-page emergency card at the front;; 2 = card;; 1 = training only;; 0 = none.

Synthesis

Controlled failure trials measure real recovery capability and turn narrow margins into improvements.

Continue studying — chapter 47 of 50

← Chapter 46 (draft) · Course chapter index · Chapter 48 (draft) →