Chapter 30 — Laboratory II
Lesson map
Lesson 30 published. Map, example, EX088.
flowchart TD
N0["Learning outcomes"]
N1["30.1 Setup"]
N2["30.2 Expectations"]
N3["30.3 Observing"]
N4["30.4 Results"]
N5["30.5 Report"]
N6["30.6 Safety"]
N7["30.7 Reflection"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX088"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["30.1 Setup"]
N2["30.2 Expectations"]
N3["30.3 Observing"]
N4["30.4 Results"]
N5["30.5 Report"]
N6["30.6 Safety"]
N7["30.7 Reflection"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX088"]
[Learning outcomes]
|
v
[30.1 Setup]
|
v
[30.2 Expectations]
|
v
[30.3 Observing]
|
v
[30.4 Results]
|
v
[30.5 Report]
|
v
[30.6 Safety]
|
v
[30.7 Reflection]
|
v
[Worked example] --> [Exercise EX088]
Learning outcomes
You will (1) run a combined verification and operations lab; (2) inject synthetic events; (3) observe detection and response; (4) report results.
30.1 Setup
In the test account, the lab injects synthetic events: a public access change, a new access key, a privileged pod request and a logging disablement attempt.
Each event is one the controls are meant to stop or catch, and each is harmless: the public change targets an empty synthetic bucket, the new key belongs to a test identity with no permissions, and the privileged pod runs a do-nothing image. The test account is separate from production, so nothing injected can reach StudyHub or student data.
30.2 Expectations
The guardrail blocks the public change and logging disablement; admission rejects the privileged pod; alerts fire for the key and the attempts.
Writing expectations before injecting anything turns the lab into a test rather than a demonstration. Each event has a predicted outcome and, where an alert is expected, a target time. If an outcome differs, students know at once that either the control or their understanding of it is wrong, and the observer records which. The expectations sheet is handed in with the report.
30.3 Observing
Students record when each event happened, whether it was blocked, when alerts arrived and whether the runbook worked.
Timings come from the observer's clock and the platform's own timestamps, which are compared afterwards. Whether the runbook worked is judged by the responder: could they confirm the alert and take the first step without asking anyone? Notes on unclear steps feed directly into runbook updates. The group also records anything surprising, such as an alert that named the wrong resource or arrived twice. Small details like these often reveal problems in alert routing that a simple pass or fail would hide.
30.4 Results
All events are blocked or detected; the new-key alert takes 18 minutes, slower than the 15-minute target, because it waits for a batch log export.
Every control worked, but the key alert's timing missed its target by a few minutes. Batch export means events are collected and sent in groups, which adds delay that real-time event streams avoid. The miss was small in this lab, but in a real key leak those minutes could matter, so it became the group's main recommendation.
30.5 Report
A short report with timings, a comparison with targets and recommendations.
Samuel's group reported each event in one row: what was injected, expected outcome, observed outcome, time to alert and target. The key alert's row was marked as missing its target. Recommendations followed from the rows, with the batch-export change first because it addressed the only failure. The report also listed what the group would test next, such as the same events in a production test namespace, and the questions they could not answer in the lab, such as whether delivery times change under heavy load. It fitted on two pages and was shared with the cloud team the same day.
30.6 Safety
Synthetic events only, in a test account, with the cloud team present.
The cloud team's presence means someone with full knowledge of the platform can stop the lab at once if anything unexpected happens. Synthetic events in a test account cannot affect production, and the clean-up step checks that nothing is left behind afterwards to confuse later checks or tests. Before the lab starts, the cloud team confirms the test account has no connections to production and that the identities used have no rights elsewhere. Students cannot inject events themselves without the coordinator, and every injection is written in the group's log with the time.
30.7 Reflection
Students discuss which detections they would trust in production.
Samuel's group trusted the guardrail blocks most, because they act before anything happens and were tested directly. They trusted the key alert less until it moved to real-time events. Students also noted that the test account may deliver logs differently from production, so production checks are needed before full trust. The discussion also covered the alerts that worked well: the logging-disablement block was immediate and clear, and students agreed it was the control they would most want in place if they could have only one. That view matched the opening case, where missing logs caused the biggest uncertainty.
Lab roles
Each group has a coordinator who injects events, an observer who records timings and a responder who follows the runbooks. Roles rotate between rounds so everyone experiences each. Having a separate observer makes the timings reliable, because the responder is busy. The cloud team sits in to answer questions about the platform and to make sure no injected event touches anything outside the test account.
Comparing groups
Groups compare their timings and runbook notes afterwards. Differences in how quickly groups confirmed alerts lead to a clearer first step in two runbooks.
Clean-up
After the lab, injected objects are removed and the test account is checked for leftovers.
Worked example
Samuel's group recommends moving key-creation alerts to the real-time event stream instead of batch export. Tom tests it the following week; alerts now arrive in under 2 minutes.
Synthetic case. Lab events are synthetic.
Errors and limits
Test accounts may differ from production in log delivery; check both.
Exercises
- EX088 — Injected events. List the injected events. Rubric: 3 = public change, new key, privileged pod, logging disable; 2 = three; 1 = one; 0 = none.
- EX089 — Slow alert. Which alert was slow and why? Rubric: 3 = new key, 18 min, batch export; 2 = partial; 1 = vague; 0 = none.
- EX090 — Fix. What fixed it? Rubric: 3 = real-time event stream; 2 = partial; 1 = vague; 0 = none.
Synthesis
Inject safe events to test detection and response end to end.
Continue studying — chapter 30 of 40
← Chapter 29 (draft) · Course chapter index · Chapter 31 (draft) →