Chapter 32 — Cloud and container security: reliability
Lesson map
Lesson 32 published. Map, example, EX094.
flowchart TD
N0["Learning outcomes"]
N1["32.1 Controls as dependencies"]
N2["32.2 High availability"]
N3["32.3 Expiry"]
N4["32.4 Timeouts"]
N5["32.5 Measuring"]
N6["32.6 Error budgets"]
N7["32.7 Game days"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX094"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["32.1 Controls as dependencies"]
N2["32.2 High availability"]
N3["32.3 Expiry"]
N4["32.4 Timeouts"]
N5["32.5 Measuring"]
N6["32.6 Error budgets"]
N7["32.7 Game days"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX094"]
[Learning outcomes]
|
v
[32.1 Controls as dependencies]
|
v
[32.2 High availability]
|
v
[32.3 Expiry]
|
v
[32.4 Timeouts]
|
v
[32.5 Measuring]
|
v
[32.6 Error budgets]
|
v
[32.7 Game days]
|
v
[Worked example] --> [Exercise EX094]
Learning outcomes
You will (1) ensure security controls do not reduce reliability; (2) design highly available control components; (3) plan for certificate and key expiry; (4) measure control-induced incidents.
32.1 Controls as dependencies
Admission controllers, policy engines, secret managers and identity services are now on the critical path for deployments and running services. If they fail, so does meterhub.
Before the project, a broken admission controller would have affected nothing because there was none. Now every deployment passes through it, every workload fetches secrets from the manager, and every cloud call uses workload identity. Security controls have become infrastructure, and must be engineered with the same care for availability as the services they protect.
32.2 High availability
The policy engine runs three replicas across zones with a disruption budget so upgrades never remove more than one. The secret manager is a managed cloud service with regional redundancy.
Three replicas across zones means the engine survives the loss of one replica or one zone. A disruption budget stops routine maintenance from removing too many at once. Using a managed secret service shifts some of the availability work to the provider, whose regional redundancy is stronger than Solent could build alone.
32.3 Expiry
Certificates and signing keys expire. Elena lists every certificate and key with expiry dates and adds alerts at 30 and 7 days. An expired webhook certificate would block all deployments.
Expiry problems are predictable, which makes outages caused by them especially frustrating. Listing every certificate and key with its date turns invisible risks into a schedule. Alerts at thirty days give time to renew calmly; seven days is the final warning. The webhook certificate is listed first because its expiry would halt all deployments.
32.4 Timeouts
Admission webhooks have short timeouts so a slow engine causes a clear failure rather than hanging deployments.
Without a timeout, a slow policy engine leaves deployments waiting indefinitely, which is harder to diagnose than an outright failure. A short timeout produces a clear error message that points to the engine. Combined with fail-closed behaviour in production, it means no workload is admitted unchecked. Elena tests the timeout by slowing the engine in staging and confirming that deployments fail within seconds with a message naming the webhook, rather than hanging until the deployment tool gives up.
32.5 Measuring
Incidents caused by security controls are tracked separately; the goal is fewer than one per quarter after rollout.
Tracking control-caused incidents separately shows whether security changes are hurting reliability. A target of fewer than one per quarter sets a clear expectation. If the count rises, the platform team investigates before adding new controls, which keeps trust with service teams. Each incident gets a short review naming the control involved, the cause and the fix. Reviews look for patterns, such as several incidents from one rule, which may mean the rule needs redesign rather than another quick patch.
32.6 Error budgets
Solent's site reliability practice gives each service an error budget; control-induced outages count against it, which gives the platform team a reason to make controls reliable.
An error budget allows a set amount of unreliability per period. When control-induced outages consume it, the platform team must spend time making controls more reliable. This gives security and reliability a shared measure, so neither is pursued at the expense of the other. Elena explains the idea to service teams with one example from the past quarter, so the budget feels concrete.
32.7 Game days
Planned exercises simulate control failures in staging to practise recovery.
Simulating failures in staging lets the team practise recovery without risking production. Scenarios include the policy engine going down and an expired webhook certificate. Each game day records how long recovery took and what was unclear, and the runbooks are updated afterwards. Each game day has a written scenario, a start time known to the participants and an observer who notes timings and confusion. Elena runs the first one with Rosa: the signing key's certificate is replaced with an expired copy in staging, and the team practises spotting the cause and rotating it from the runbook.
Disruption budgets
A pod disruption budget tells Kubernetes how many replicas of a component may be unavailable during voluntary disruptions such as node upgrades. For the policy engine it is set to allow at most one unavailable replica. Elena tests it by draining a node in staging and confirming the engine keeps serving admission requests throughout. Similar budgets are set for ingest, so upgrades never drop it below the capacity needed for the evening peak.
Worked example
A game day simulates the signing key's certificate expiring. Deployments are blocked as designed; the runbook for emergency rotation takes 25 minutes, longer than the 10-minute target. Steps are scripted and the next exercise completes in 8 minutes.
Synthetic case. Components and timings are synthetic.
Errors and limits
Reliability work on controls is often skipped because controls are seen as "security", not "service"; they are both.
Exercises
- EX094 — Critical path. Why are controls on the critical path? Rubric: 3 = deployments and services depend on them; 2 = partial; 1 = vague; 0 = none.
- EX095 — Expiry alerts. When do expiry alerts fire? Rubric: 3 = 30 and 7 days; 2 = partial; 1 = vague; 0 = none.
- EX096 — Game day result. What did the game day show? Rubric: 3 = rotation 25 min, then 8 after scripting; 2 = partial; 1 = vague; 0 = none.
Synthesis
Make security controls as reliable as the services they protect.
Continue studying — chapter 32 of 50
← Chapter 31 (draft) · Course chapter index · Chapter 33 (draft) →