Chapter 33 — Cloud and container security: safety and resilience
Lesson map
Lesson 33 published. Map, example, EX097.
flowchart TD
N0["Learning outcomes"]
N1["33.1 Essential functions"]
N2["33.2 Graceful degradation"]
N3["33.3 Recovery from compromise"]
N4["33.4 Backups and immutability"]
N5["33.5 Isolation of blast radius"]
N6["33.6 Exercises"]
N7["33.7 Communication"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX097"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["33.1 Essential functions"]
N2["33.2 Graceful degradation"]
N3["33.3 Recovery from compromise"]
N4["33.4 Backups and immutability"]
N5["33.5 Isolation of blast radius"]
N6["33.6 Exercises"]
N7["33.7 Communication"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX097"]
[Learning outcomes]
|
v
[33.1 Essential functions]
|
v
[33.2 Graceful degradation]
|
v
[33.3 Recovery from compromise]
|
v
[33.4 Backups and immutability]
|
v
[33.5 Isolation of blast radius]
|
v
[33.6 Exercises]
|
v
[33.7 Communication]
|
v
[Worked example] --> [Exercise EX097]
Learning outcomes
You will (1) design meterhub to keep essential functions during incidents; (2) separate safety-relevant data paths; (3) plan recovery from compromise; (4) test resilience.
33.1 Essential functions
For an energy co-op, essential functions are ingesting readings (data must not be lost), billing accuracy and tamper alerts. Customer portal features are less critical.
Ranking functions tells engineers what to protect first when resources are stretched. Readings that are lost cannot be recollected, so ingestion comes first. Billing accuracy matters for members' trust and regulatory compliance. Tamper alerts protect the grid. Portal features can wait if necessary. Elena writes the ranking into the incident plan, so responders know which services to restore first and which can stay down a little longer while attention goes elsewhere.
33.2 Graceful degradation
If the API is attacked or overloaded, ingest continues on separate infrastructure. Queues buffer readings if the database is unavailable.
Separating ingest from the customer-facing API means an attack on the portal does not stop meters reporting. Queues hold readings while the database is down and deliver them when it returns, so no data are lost. Elena checks that the queue's capacity covers the longest expected database outage. She also checks that ingest uses no component shared with the API, such as a common database connection pool, which could let the overload spread.
33.3 Recovery from compromise
If a cluster is suspected compromised, the plan is to rebuild from code in a clean account rather than clean it in place: declarative configuration and signed images make rebuilds repeatable.
Cleaning a compromised cluster in place risks missing a hidden change. Rebuilding from reviewed code in a fresh account starts from a known state. Signed images ensure only Solent-built software runs. The approach depends on everything being defined in code, which earlier chapters made true. The rebuild runbook is tested in the resilience exercises.
33.4 Backups and immutability
Backups in a separate account with deletion protection allow recovery even if production credentials are misused.
Deletion protection stops anyone, including an attacker with production credentials, from removing backups before they expire. A separate account adds another barrier. Together they ensure that even a severe compromise of production leaves a clean copy to restore from. Restores from these backups are tested quarterly, and the restored data are compared with production counts to confirm nothing is missing. Retention on the backups is longer than any plausible delay in discovering a compromise, so a clean copy predates the attacker's arrival.
33.5 Isolation of blast radius
Separate clusters and accounts (chapter 26) limit how far a compromise spreads.
Separating the high-sensitivity cluster from the standard one means a compromise of a general service cannot directly reach readings or payments. Separate accounts for backups and logs add further walls. Each boundary limits how far an incident can spread and simplifies containment. Elena lists each boundary with the credentials needed to cross it, which shows that no single identity can reach both clusters, the backup account and the log account. The list is checked whenever a new role is created, so the boundaries stay intact as the platform grows.
33.6 Exercises
Resilience exercises rebuild the standard cluster from code in a fresh account and measure the time to restore ingest.
Rebuilding from code proves that documentation and templates are complete enough to recreate the platform. Measuring time to restore ingest shows whether the recovery objective is realistic. Each exercise finds small gaps, such as a missing permission, which are fixed before the next. The first exercise shows which steps still wait on people, and those steps are the first to be automated or pre-approved. Results are reported to Solent's risk committee with the list of fixes.
33.7 Communication
Incident plans include messages to customers and the regulator.
Customers deserve to know if their data might be affected, and the regulator expects prompt notification. Pre-written messages save time during an incident and reduce the risk of saying something inaccurate. The data protection officer approves the templates in advance. The plan names who sends each message, who approves it and the deadline that applies, including any legal notification period. Messages avoid technical jargon and say what members should do, if anything. The templates are reviewed once a year and after any real incident, and the contact list for the regulator is checked at the same time.
Tamper alerts
Tamper alerts from meters are routed through a separate, small service with its own queue, so they keep flowing even if the main ingest path is degraded. Elena includes this path in the rebuild exercise and confirms alerts resume within 20 minutes in a fresh account. She also notes that tamper alerts have a regulatory reporting duty, so their path is documented for the regulator.
Recording lessons
Each exercise ends with a short review listing what slowed recovery and the fix.
Worked example
The first rebuild exercise takes 6 hours, mainly waiting for manual approvals to create new accounts. Pre-approved account templates cut the next exercise to 2.5 hours, within the 4-hour objective for restoring ingest.
Synthetic case. Exercises and timings are synthetic.
Errors and limits
Rebuild plans fail if secrets or configuration live outside version control; inventory them.
Exercises
- EX097 — Essential functions. List meterhub's essential functions. Rubric: 3 = ingest, billing accuracy, tamper alerts; 2 = two; 1 = one; 0 = none.
- EX098 — Rebuild reason. Why rebuild rather than clean a compromised cluster? Rubric: 3 = repeatable from code and signed images; 2 = partial; 1 = vague; 0 = none.
- EX099 — Exercise time. How long did rebuilds take? Rubric: 3 = 6 h then 2.5 h; 2 = partial; 1 = vague; 0 = none.
Synthesis
Plan to keep essential functions and rebuild cleanly.
Continue studying — chapter 33 of 50
← Chapter 32 (draft) · Course chapter index · Chapter 34 (draft) →