Chapter 29 — Cloud and container security: operations
Lesson map
Lesson 29 published. Map, example, EX085.
flowchart TD
N0["Learning outcomes"]
N1["29.1 Daily work"]
N2["Lab — Lab — alert handling"]
N3["29.2 Lab result"]
N4["29.3 Tuning"]
N5["29.4 Runbooks"]
N6["29.5 Measuring"]
N7["29.6 Workload"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX085"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["29.1 Daily work"]
N2["Lab — Lab — alert handling"]
N3["29.2 Lab result"]
N4["29.3 Tuning"]
N5["29.4 Runbooks"]
N6["29.5 Measuring"]
N7["29.6 Workload"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX085"]
[Learning outcomes]
|
v
[29.1 Daily work]
|
v
[Lab — Lab — alert handling]
|
v
[29.2 Lab result]
|
v
[29.3 Tuning]
|
v
[29.4 Runbooks]
|
v
[29.5 Measuring]
|
v
[29.6 Workload]
|
v
[Worked example] --> [Exercise EX085]
Learning outcomes
You will (1) run day-to-day cloud security operations; (2) measure alert quality and response time; (3) tune alerts; (4) keep runbooks current.
29.1 Daily work
The cloud team reviews alerts, check results and exception expiries daily.
The review takes about half an hour each morning and is done by whoever is on duty that week. Alerts are checked first, since they may need immediate action; check results next, since they show what drifted overnight; and exception expiries last, so temporary permissions are renewed deliberately or allowed to lapse. Anything that cannot be finished is written into the handover note described below, with an owner and the next step, so the work is never lost between people.
Lab — Lab — alert handling
"""Summarise synthetic alert handling: count, actionable share and median minutes to acknowledge (stdlib only)."""
import statistics
alerts = [("public-access-change", True, 6), ("public-access-change", True, 9), ("new-access-key", True, 14),
("new-access-key", False, 30), ("login-new-country", False, 45), ("login-new-country", False, 60),
("login-new-country", False, 38), ("logging-disabled", True, 4)]
rules = sorted({a[0] for a in alerts})
for r in rules:
x = [a for a in alerts if a[0] == r]
print(f"{r:21} n {len(x)}, actionable {sum(a[1] for a in x)}/{len(x)}, median ack {statistics.median(a[2] for a in x)} min")
Run with python3 lab_alerts.py. Verified output:
logging-disabled n 1, actionable 1/1, median ack 4 min
login-new-country n 3, actionable 0/3, median ack 45 min
new-access-key n 2, actionable 1/2, median ack 22.0 min
public-access-change n 2, actionable 2/2, median ack 7.5 min
29.2 Lab result
logging-disabled: 1 alert, 1 actionable, median 4 minutes to acknowledge. public-access-change: 2 alerts, both actionable, median 7.5 minutes. new-access-key: 2 alerts, 1 actionable, median 22.0 minutes. login-new-country: 3 alerts, none actionable, median 45 minutes. Alerts that are never actionable get slow responses, which risks slow responses to real ones too.
The pattern is clear in the numbers: rules whose alerts always need action are answered within minutes, while the rule whose alerts never need action waits the better part of an hour. People learn quickly which alerts matter. The danger is that a real login from an attacker's country would wait just as long as the conference trips, because the rule has taught the team to treat it as noise.
29.3 Tuning
login-new-country fires for staff at conferences; it is tuned to exclude known travel from the identity provider's records and to raise severity only for administrator roles.
Using the identity provider's travel records means the rule no longer needs to guess whether a login abroad is expected. Raising severity only for administrator roles focuses attention where a stolen session would do most damage. The tuned rule still records every login from a new country, so the information remains available for investigations even when no alert is raised.
29.4 Runbooks
Each alert links to a runbook; runbooks are updated after each real alert.
Linking the runbook from the alert itself means the person on duty does not have to search for it at the moment they need it. Updating after each real alert keeps the runbook matched to how the platform actually behaves, rather than how it behaved when the runbook was written. Samuel's review of all runbooks, described below, found several that had drifted. Each update is reviewed by a second team member.
29.5 Measuring
Monthly: alerts by rule, actionable share and acknowledgement time.
These three numbers show whether alerts are worth the team's time. A rule whose actionable share stays near zero is a candidate for tuning or removal; a rising acknowledgement time suggests overload or unclear runbooks. Samuel adds each month's figures to a small chart, so the effect of the login-new-country tuning is visible as a step down in total alerts. The figures are shared with Tom at the monthly register review, so decisions about tuning and staffing rest on the same numbers everyone can see.
29.6 Workload
Tuning keeps total alerts manageable for a small team.
The cloud team has a handful of people who also build and run StudyHub, so every unnecessary alert takes time from other work. After tuning, the team receives a few alerts a week instead of several a day, and almost all of them need action. That keeps attention high, which is the point of having alerts at all. Samuel also notes that tuning is revisited each term, since StudyHub changes and a rule that was quiet may become noisy, or a tuned rule may start missing something real.
Runbook quality
A runbook states what the alert means, how to check whether it is real, what to do first, who to tell and how to close it. Runbooks are short enough to read during an incident. After each real alert, the responder notes what was unclear, and the runbook is updated within a week. Samuel reviews all runbooks once and finds three still referring to an old ticket system, which are corrected. He also adds a 'last tested' date to each runbook, so stale ones are easy to spot.
Handover notes
At the end of each day, the person on duty writes a short note of open alerts and actions so the next person starts with full context. Notes are kept for a month.
Worked example
After tuning, login-new-country alerts drop from about 40 a month to 3, and median acknowledgement for all alerts falls below 10 minutes.
Synthetic case. Alerts are synthetic.
Errors and limits
Tuning can hide real events; review tuned rules periodically.
Exercises
- EX085 — Lab worst rule. Which rule was least actionable? Rubric: 3 = login-new-country 0/3; 2 = partial; 1 = vague; 0 = none.
- EX086 — Why it matters. Why do non-actionable alerts matter? Rubric: 3 = slow responses spread; 2 = partial; 1 = vague; 0 = none.
- EX087 — Tuning result. What did tuning achieve? Rubric: 3 = 40 to 3 a month; ack under 10 min; 2 = partial; 1 = vague; 0 = none.
Synthesis
Measure alert quality and tune for action.
Continue studying — chapter 29 of 40
← Chapter 28 (draft) · Course chapter index · Chapter 30 (draft) →