Chapter 28 — Cloud and container security: failure analysis
Lesson map
Lesson 28 published. Map, example, EX082.
flowchart TD
N0["Learning outcomes"]
N1["28.1 Approach"]
N2["28.2 Admission control failure"]
N3["28.3 Identity provider outage"]
N4["28.4 Logging failure"]
N5["28.5 Human failures"]
N6["28.6 Prioritising"]
N7["28.7 Exercises"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX082"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["28.1 Approach"]
N2["28.2 Admission control failure"]
N3["28.3 Identity provider outage"]
N4["28.4 Logging failure"]
N5["28.5 Human failures"]
N6["28.6 Prioritising"]
N7["28.7 Exercises"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX082"]
[Learning outcomes]
|
v
[28.1 Approach]
|
v
[28.2 Admission control failure]
|
v
[28.3 Identity provider outage]
|
v
[28.4 Logging failure]
|
v
[28.5 Human failures]
|
v
[28.6 Prioritising]
|
v
[28.7 Exercises]
|
v
[Worked example] --> [Exercise EX082]
Learning outcomes
You will (1) analyse how StudyHub controls could fail; (2) consider technical and human failures; (3) plan detection and recovery for each; (4) prioritise improvements.
28.1 Approach
For each important control, ask how it could fail, how failure would be noticed and how service would recover.
The three questions turn a list of controls into a list of risks. For the registry, failure means images become public again; noticing it depends on the alert for public settings; recovery means reverting the change and checking pull logs. Asking all three for every control finds the silent ones. Samuel works through the register one control at a time. Answers go into the failure table.
28.2 Admission control failure
If the admission service is down, deployments may be blocked (fail closed) or allowed without checks (fail open). Westmere chooses fail closed for production with alerts and a runbook.
Fail closed means a broken admission service stops deployments rather than letting unchecked pods through. That protects students' data but can delay urgent fixes, so the runbook explains how to restore the admission service quickly. The alert ensures someone knows within minutes. Test clusters fail open to keep development moving, which is acceptable because they hold no student data.
28.3 Identity provider outage
Staff cannot sign in; break-glass accounts allow emergency access.
Without break-glass accounts, an identity provider outage would lock staff out of the cloud console at exactly the moment they might need it. The accounts' sealed credentials and immediate alerts on use keep them from becoming a back door. Their twice-yearly test confirms they still work. Using them also raises an alert, so an outage is never quietly used as cover for unreviewed changes. After each use, the credentials are resealed and the use is recorded in the access review, with who used them and why.
28.4 Logging failure
If log delivery stops, an alert fires when no logs arrive for 30 minutes.
Thirty minutes is short enough that a gap in evidence stays small and long enough to avoid alerts from normal delivery delays. The alert goes to the cloud team's on-call rota. The opening case shows why this matters: a logging gap that nobody noticed made the incident impossible to measure. The alert checks the central log account for each source, so a gap in one account is noticed even when others are still delivering. Samuel tested it by pausing delivery from the test account and confirming the alert fired on time.
28.5 Human failures
A rushed console change during an incident; a mistaken guardrail exception. Controls: changes through code where possible, exceptions with expiry.
During an incident, pressure tempts people to click through console changes that bypass review. Making changes through code, even urgent ones, keeps a record and allows a quick second look. Exceptions with expiry dates ensure that a temporary guardrail relaxation does not quietly become permanent. Samuel's register records each guardrail exception with its owner and expiry date. A weekly check lists exceptions due to expire, so owners can renew them deliberately or let them lapse. In the first month, two expired exceptions were found still in place and removed.
28.6 Prioritising
Failures that would be silent rank highest, because they can last long unnoticed, as in the opening case.
In the opening case, the missing logs were a silent failure: nothing alerted, so nobody knew how long the registry was open. Loud failures, such as blocked deployments, get fixed quickly because people complain. Ranking silent failures first targets the ones most likely to persist. The failure table marks each failure as loud or silent, and silent ones receive a detection check before anything else. For the secret scanner, that meant making its crash fail the build, as the worked example below shows.
28.7 Exercises
Failure scenarios are rehearsed in the test environment each term.
Each rehearsal picks one scenario from the failure table, such as the admission service going down, and the team follows the runbook in the test environment. Timings and problems are recorded, and the runbook is updated. Rehearsals have twice found steps that no longer matched the current setup. Rehearsals are short, usually under an hour, so they can run during normal working time. Each one ends with a note of what went well, what did not, and changes made. The note is linked from the failure table, so the next person to rehearse the same scenario starts from the latest lessons.
Failure table
The analysis is kept as a table: control, failure mode, how it would be noticed, recovery steps and owner. Silent failure modes are highlighted. The table itself is reviewed each term and after incidents. Samuel finds that writing the 'how would we notice' column was the most revealing part, because for four failure modes the honest answer was 'we would not', and each of those became an improvement item with a date.
Owners
Each failure mode has an owner who checks its detection works.
Worked example
The analysis finds that if the secret scanner in the pipeline crashed, builds would continue without scanning and nobody would notice. The pipeline is changed so that a scanner error fails the build, and a weekly report shows scan counts.
Synthetic case. Scenarios are synthetic.
Errors and limits
Fail-closed choices create availability risks; make the control itself reliable.
Exercises
- EX082 — Fail closed. What does fail closed mean for admission? Rubric: 3 = deployments blocked if service down; 2 = partial; 1 = vague; 0 = none.
- EX083 — Silent failures. Why do silent failures rank highest? Rubric: 3 = last long unnoticed; 2 = partial; 1 = vague; 0 = none.
- EX084 — Scanner fix. How was the scanner failure addressed? Rubric: 3 = scanner error fails build; weekly counts; 2 = partial; 1 = vague; 0 = none.
Synthesis
Plan for controls failing, especially silently.
Continue studying — chapter 28 of 40
← Chapter 27 (draft) · Course chapter index · Chapter 29 (draft) →