Chapter 34 — Comparative project
Lesson map
Lesson 34 published. Map, example, EX100.
flowchart TD
N0["Learning outcomes"]
N1["34.1 The problem"]
N2["34.2 Criteria"]
N3["34.3 Testing"]
N4["34.4 Comparison"]
N5["34.5 Recommendation"]
N6["34.6 Cost"]
N7["Writing the comparison"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX100"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["34.1 The problem"]
N2["34.2 Criteria"]
N3["34.3 Testing"]
N4["34.4 Comparison"]
N5["34.5 Recommendation"]
N6["34.6 Cost"]
N7["Writing the comparison"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX100"]
[Learning outcomes]
|
v
[34.1 The problem]
|
v
[34.2 Criteria]
|
v
[34.3 Testing]
|
v
[34.4 Comparison]
|
v
[34.5 Recommendation]
|
v
[34.6 Cost]
|
v
[Writing the comparison]
|
v
[Worked example] --> [Exercise EX100]
Learning outcomes
You will (1) compare two approaches to a cloud security problem; (2) define criteria; (3) test both on synthetic data; (4) recommend with evidence.
34.1 The problem
How should Westmere detect public storage and registry exposure: periodic scanning of configuration, or real-time alerts on configuration change events?
Both approaches already exist in some form at Westmere: Samuel's weekly checker is a periodic scan, and the alert for public registry settings is event-based. The question is which to rely on, and the comparison tests both on the same synthetic events so the answer rests on evidence rather than preference. Samuel writes the question and criteria down before building the test, and Tom agrees them.
34.2 Criteria
Time to detect, false alerts, cost, coverage of resources created outside code, and effort to maintain.
Each criterion reflects a real concern. Time to detect matters because exposure grows with every hour; false alerts consume the small team's attention; cost must fit the budget; coverage of resources created outside code matters because console changes bypass review; and maintenance effort decides whether the approach will still work in a year. Samuel weights none of them in advance. Results speak for themselves.
34.3 Testing
In a test account, Samuel creates 20 synthetic exposure events at random times. Periodic scanning every 24 hours detects all 20 with a median delay of 11 hours. Event-based alerts detect 19 with a median delay of 2 minutes; one is missed because that resource type does not emit the event used.
Creating events at random times avoids flattering either approach, for example by placing all events just before a scheduled scan. Twenty events give a usable comparison while staying quick to run. The missed event shows the value of testing: nobody had known that resource type did not emit the expected event.
34.4 Comparison
Event alerts are much faster but miss some types; scanning is complete but slow.
The two approaches fail in different ways. Event alerts depend on every resource type emitting the right event, which one did not; scanning depends only on reading configuration, so it catches everything eventually. Their weaknesses do not overlap, which is why combining them works better than either alone. Speed matters most for exposures that an outsider could find quickly, such as a public bucket; completeness matters most for rare resource types that event alerts might not cover. A table showing both lets the team see which approach protects against which kind of risk.
34.5 Recommendation
Use both: event alerts for speed, daily scanning as a safety net, plus prevention through guardrails so most events never happen.
Guardrails come first in the recommendation because an exposure that never happens needs no detection. Event alerts then catch what slips through within minutes, and daily scanning catches anything the alerts miss, such as the resource type found in testing. Each layer covers a gap in the one before. The recommendation also names an owner for each layer: the platform team for guardrails, the cloud team for alerts and Samuel's checker, later handed over, for scanning. Each layer is in the control register with a test.
34.6 Cost
Both use built-in services; cost is mainly setup time.
Event alerts and scanning both use features Westmere already pays for, so the main cost is the engineer time to set them up and keep them working. Samuel estimated a few days for setup and a small amount of monthly maintenance, which Tom accepted without needing extra budget. The test scripts and synthetic events are kept, so repeating the comparison after a platform change costs only an afternoon. Samuel notes that the cheapest option of all, relying on guardrails alone, was rejected because guardrails cannot cover every resource type or every console change.
Writing the comparison
The report presents a table of criteria with results for each approach and a short paragraph on each criterion. Samuel avoids declaring an overall winner, since the approaches complement each other; instead he shows what each contributes. Tom finds this more useful than a single recommendation, because it explains why both are kept and which one to rely on for each kind of risk.
Repeat plan
The comparison will be repeated after any change to logging or event sources.
Limits noted
The test account lacked some production services.
Worked example
The missed event type is reported to the cloud team, who add a second event source for it. A rerun detects all 20 events in under 5 minutes.
Synthetic case. Tests and figures are synthetic.
Errors and limits
Test accounts may not represent all production behaviours.
Exercises
- EX100 — Two approaches. What two approaches were compared? Rubric: 3 = periodic scanning vs event alerts; 2 = partial; 1 = vague; 0 = none.
- EX101 — Results. Give the median delays. Rubric: 3 = scan 11 h; events 2 min; 2 = partial; 1 = vague; 0 = none.
- EX102 — Recommendation. What was recommended? Rubric: 3 = both, plus guardrails; 2 = partial; 1 = vague; 0 = none.
Synthesis
Compare approaches on clear criteria with tests.
Continue studying — chapter 34 of 40
← Chapter 33 (draft) · Course chapter index · Chapter 35 (draft) →