Chapter 37 — Cloud and container security: operations
Lesson map
Lesson 37 published. Map, example, EX109.
flowchart TD
N0["Learning outcomes"]
N1["37.1 Daily operations"]
N2["37.2 Upgrades"]
N3["37.3 Exception handling"]
N4["37.4 Runbooks"]
N5["37.5 Operational load"]
N6["37.6 Incident example"]
N7["37.7 Continuous improvement"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX109"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["37.1 Daily operations"]
N2["37.2 Upgrades"]
N3["37.3 Exception handling"]
N4["37.4 Runbooks"]
N5["37.5 Operational load"]
N6["37.6 Incident example"]
N7["37.7 Continuous improvement"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX109"]
[Learning outcomes]
|
v
[37.1 Daily operations]
|
v
[37.2 Upgrades]
|
v
[37.3 Exception handling]
|
v
[37.4 Runbooks]
|
v
[37.5 Operational load]
|
v
[37.6 Incident example]
|
v
[37.7 Continuous improvement]
|
v
[Worked example] --> [Exercise EX109]
Learning outcomes
You will (1) run the secured platform day to day; (2) handle alerts, exceptions and upgrades; (3) keep runbooks current; (4) measure operational load.
37.1 Daily operations
The on-call engineer reviews overnight alerts, the exception expiry list and the drift report. Most days take under 30 minutes.
The order is deliberate. Alerts come first because one might describe something still happening. Expiries come next, since an exception that lapses unnoticed can block a deployment or quietly be renewed without review. The drift report comes last: it lists resources whose live settings differ from the declared configuration, such as a CPU limit raised by hand during a busy evening. A day that runs much longer than half an hour is noted, because it usually points at a noisy rule.
37.2 Upgrades
Kubernetes minor versions are upgraded within the provider's support window. Each upgrade is tested in staging with the verification suite, since admission behaviour can change between versions.
A new version may deprecate an API that a policy relies on, change a default for pod security, or alter how webhooks are called during admission. Running the verification suite in staging catches these before production does. Waiting until the support window is about to close is risky, because security fixes stop and several versions may then need to be crossed in a hurry. Elena plans upgrades a quarter ahead and records the suite's results with each one.
37.3 Exception handling
Weekly, the platform lead reviews new requests and upcoming expiries. Exceptions near expiry trigger a message to owners.
Each request must state the control being relaxed, the reason, the compensating measure, an owner and an end date. The review looks for patterns as well as individual cases: several services asking for the same exception usually means the platform is missing something, not that every team is careless. When an exception reaches its end date without renewal, the control applies again automatically, so a forgotten exception fails safe instead of lasting for ever.
37.4 Runbooks
Each alert has a runbook with first steps, escalation and communication. Runbooks are tested in game days and updated after incidents.
A good first step is concrete, for example 'check whether the denied image digest appears in the signing log', rather than 'investigate'. Escalation names a role, not a person, so the runbook survives staff changes. Each runbook records the date it was last exercised. During a game day an engineer who has never handled that alert follows the runbook without help, and every point where they hesitate becomes an edit.
37.5 Operational load
Elena tracks time spent on security operations: in the first quarter after rollout, about 6 engineer-hours a week, falling to 3 as tooling improves.
The hours are logged by category: alert handling, exception reviews, upgrades and drift fixes. The breakdown shows where tooling helps most. In this case much of the early time went on repeated exception renewals and alerts that needed manual lookups, both of which shrank once the underlying needs were removed and the lookups were added to the alerts. A rise in hours is treated as an early warning, since overloaded teams tend to relax controls to save time.
37.6 Incident example
A developer accidentally creates a public test bucket in a sandbox account. The account-level block prevents public access; an alert still fires for the attempted change, and a short note reminds the developer of the sandbox rules.
The sequence shows layered controls working as designed: the preventive block stopped exposure, and the detective alert made sure someone knew it had been tried. The response is proportionate, with no data at risk and no blame, only a reminder. Elena still records it, because a cluster of similar attempts from one team would suggest that a legitimate need, such as sharing test files with a partner, is not being met in a safe way.
37.7 Continuous improvement
Each month the team picks one source of repetitive work and automates or removes it.
Choosing one item a month keeps the effort steady and small enough to finish. Candidates come from the operational-load log and from handover notes, where the same annoyance tends to appear week after week. The repeated exceptions for a log path written to the root filesystem are a typical case: fixing the image removed both the exceptions and the review work they caused. Each change is recorded with the hours it saved, so the team can see the effect.
Handover between shifts
On-call handover includes a short written note: open alerts, exceptions expiring that week, upgrades planned and anything unusual. The note lives in the team channel so others can see the platform's state at a glance. Missed handovers were a cause of two slow responses before this practice started.
Calm operations
A quiet platform is the goal.
Worked example
Exception reviews show three services renewing the same exception repeatedly. The platform team works with them to remove the underlying need, a log path written to the root filesystem, and the exceptions end.
Synthetic case. Operations data are synthetic.
Errors and limits
Operational load that grows unnoticed leads to controls being weakened; track it.
Exercises
- EX109 — Daily tasks. What does on-call review daily? Rubric: 3 = alerts, exception expiries, drift report; 2 = two; 1 = one; 0 = none.
- EX110 — Upgrade testing. Why test upgrades with the verification suite? Rubric: 3 = admission behaviour can change; 2 = partial; 1 = vague; 0 = none.
- EX111 — Repeated exceptions. How were repeated exceptions ended? Rubric: 3 = removed underlying need; 2 = partial; 1 = vague; 0 = none.
Synthesis
Steady operations keep controls effective without exhausting teams.
Continue studying — chapter 37 of 50
← Chapter 36 (draft) · Course chapter index · Chapter 38 (draft) →