Chapter 33 — Systems, endpoints and hardware security: safety and resilience
Lesson map
Lesson 33 published. Map, example, EX097.
flowchart TD
N0["Learning outcomes"]
N1["33.1 Bad image"]
N2["33.2 Firmware updates"]
N3["33.3 Server outage"]
N4["33.4 Recovery drills"]
N5["33.5 Human factors"]
N6["Fail-safe design"]
N7["Documentation for recovery"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX097"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["33.1 Bad image"]
N2["33.2 Firmware updates"]
N3["33.3 Server outage"]
N4["33.4 Recovery drills"]
N5["33.5 Human factors"]
N6["Fail-safe design"]
N7["Documentation for recovery"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX097"]
[Learning outcomes]
|
v
[33.1 Bad image]
|
v
[33.2 Firmware updates]
|
v
[33.3 Server outage]
|
v
[33.4 Recovery drills]
|
v
[33.5 Human factors]
|
v
[Fail-safe design]
|
v
[Documentation for recovery]
|
v
[Worked example] --> [Exercise EX097]
Learning outcomes
You will (1) plan for failures of the design itself; (2) recover from bad images and firmware updates; (3) test recovery.
33.1 Bad image
If a released image breaks PCs, staged release limits impact, and PCs can roll back to the previous signed image.
Staged release means a bad image reaches the lab and one branch before the rest, so a problem found there affects a few dozen PCs at most. Rollback uses the previous signed image, which each PC keeps until the new one has run successfully for a week. Mateo tested rollback on lab PCs with a deliberately faulty synthetic image, and every PC returned to the earlier release at the next restore.
33.2 Firmware updates
NIST SP 800-193 emphasises recovery for firmware. Fernhill keeps vendor recovery tools and tests firmware updates on lab PCs first.
Firmware updates carry more risk than image updates, because a failed one can leave a PC unable to start. Mateo tests each update on a lab PC of the same model, checks that settings such as Secure Boot and the firmware password survive, and keeps the vendor's recovery tool and a known-good firmware file ready before rolling out. Updates go to branches a few PCs at a time.
33.3 Server outage
PCs restore from local protected copies for up to a week without the server.
The local copy came from the first design studio, where a guest asked what would happen if the server failed. A week covers a long outage without forcing PCs out of service. If the server stays down longer, PCs keep restoring from the same copy, but new image releases and their updates stop arriving, so the risk grows over time. The dashboard shows when each PC last reached the server.
33.4 Recovery drills
Each quarter, the team practises rolling back an image on lab PCs.
A drill uses a synthetic faulty image on lab PCs and times each step: noticing the problem, deciding to roll back, carrying out the rollback and confirming the PCs are healthy. The worked example shows what drills reveal. Steps that take longer than expected are improved before the next drill. Records of each drill are kept for audit and to show whether the team is getting faster. Marta reviews every record.
33.5 Human factors
Instructions are short and printed, for use under pressure.
Mateo wrote the procedures for a technician working alone early in the morning with patrons waiting outside. Each step fits on one line, with the expected result beside it. A printed copy sits in each branch's IT cupboard as well as online, since an outage may also affect access to the online version. Librarians tested the instructions and found two unclear steps, which were rewritten before the first drill. Each copy carries a date.
Fail-safe design
When something goes wrong, the PC should fail in a safe state. An out-of-service screen is safe: no patron uses a PC in an unknown state. Opening a session on a PC whose restore failed would be unsafe, because the previous patron's files might remain. Saltzer and Schroeder (1975) called this fail-safe defaults: deny by default when uncertain.
Mateo applies the same rule to updates: a PC that cannot verify a new image's signature keeps its current image.
Documentation for recovery
Each recovery procedure lists prerequisites, such as keys and tools, so they are prepared in advance.
Test cadence
Recovery tests run quarterly and after any major change.
Shared responsibility for recovery
Vendors supply firmware recovery tools; Fernhill keeps them ready.
People in recovery
At least two trained people can perform each recovery step.
Communication during recovery
Librarians receive a short message explaining which PCs are affected and when they will return.
Approval
Recovery plans are approved.
Worked example
A drill reveals that rollback requires a key held only by Marta. A second authorised person is added.
Synthetic case. Synthetic.
Errors and limits
Recovery that depends on one person is fragile.
Exercises
- EX097 — Bad image. How is a bad image handled? Rubric: 3 = staged release, rollback to previous signed; 2 = partial; 1 = vague; 0 = none.
- EX098 — Server outage. How long can PCs work without the server? Rubric: 3 = up to a week; 2 = partial; 1 = vague; 0 = none.
- EX099 — Drill finding. What did the drill reveal? Rubric: 3 = key only with Marta; 2 = partial; 1 = vague; 0 = none.
Synthesis
Design recovery for the design's own failures.
Continue studying — chapter 33 of 50
← Chapter 32 (draft) · Course chapter index · Chapter 34 (draft) →