Chapter 23 — Cloud and container security: evaluation
Lesson map
Lesson 23 published. Map, example, EX067.
flowchart TD
N0["Learning outcomes"]
N1["23.1 What to compare"]
N2["Lab — before and after"]
N3["23.2 Reading the result"]
N4["23.3 Checking the advisory time"]
N5["23.5 Remaining items"]
N6["Sharing results"]
N7["Honest framing"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX067"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["23.1 What to compare"]
N2["Lab — before and after"]
N3["23.2 Reading the result"]
N4["23.3 Checking the advisory time"]
N5["23.5 Remaining items"]
N6["Sharing results"]
N7["Honest framing"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX067"]
[Learning outcomes]
|
v
[23.1 What to compare]
|
v
[Lab — before and after]
|
v
[23.2 Reading the result]
|
v
[23.3 Checking the advisory time]
|
v
[23.5 Remaining items]
|
v
[Sharing results]
|
v
[Honest framing]
|
v
[Worked example] --> [Exercise EX067]
Learning outcomes
You will (1) compare measures before and after improvements; (2) judge whether changes worked; (3) state limits honestly.
23.1 What to compare
Priya returns to her measures from chapter 9: unapproved images, baseline failures and time to answer an advisory. Using the same definitions matters: unapproved images are still counted against the full image references on the approved list, and baseline failures are still counted on the same six checks. The advisory time is measured on a question Priya has not seen before, following Kwame's rule in the measurement plan. Changing any definition halfway would make the before-and-after comparison meaningless. Priya runs each measure exactly as she did at the start of the placement.
Lab — before and after
"""Compare unapproved images and baseline failures before and after the new checks (synthetic)."""
before = {"unapproved_images": 3, "baseline_fails": 7, "hours_to_answer_advisory": 6.0}
after = {"unapproved_images": 1, "baseline_fails": 2, "hours_to_answer_advisory": 0.5}
for k in before:
b, a = before[k], after[k]
print(f"{k}: {b} -> {a} ({(b-a)/b*100:.0f}% lower)")
Run with python3 lab_compare.py. Verified output:
unapproved_images: 3 -> 1 (67% lower)
baseline_fails: 7 -> 2 (71% lower)
hours_to_answer_advisory: 6.0 -> 0.5 (92% lower)
23.2 Reading the result
Unapproved images fall from 3 to 1 (67% lower), baseline failures from 7 to 2 (71% lower), and the time to answer an advisory from 6 hours to half an hour (92% lower). In counts, two unapproved images and five baseline failures have been resolved. The percentages look large because the starting counts are small, so Priya presents the counts first, as she does with every measure. The advisory time is the most direct answer to the placement trigger, since it measures the very question that once took the team six hours.
23.3 Checking the advisory time
To measure fairly, Kwame gives Priya a new synthetic advisory without warning. She answers it from the inventory in about 30 minutes, including checking two edge cases. The two edge cases were a workload whose image had been rebuilt the previous day and one in a namespace that had been renamed, both of which needed a check against the cluster rather than the inventory alone. Timing ran from receiving the advisory to sending Kwame the list of affected workloads, so it included the checks and the write-up.
23.4 Limits
The counts are small and come from one placement period. Other changes, such as staff leaving, also affected results. Staff departures, for instance, removed some accounts and workloads during the same period, which may have lowered counts independently of Priya's checks. The advisory test used one synthetic advisory, so the half-hour figure shows what is possible rather than a reliable average. Priya states these limits on the same slide as the results, not at the end. Kwame asks her to keep these limits in the client version of the results too.
23.5 Remaining items
One unapproved image (cron, waiting for the partner's new format) and two baseline failures (internal cluster default and client-a endpoint during change window) remain, each with an owner and date. Each remaining item has a reason that is not neglect. Cron waits on a partner's format change before it can move to an approved image; the internal cluster's default stays at Baseline until warn mode clears its workloads; and client-a's endpoint change needs an agreed window. Recording the reasons keeps the items from looking like forgotten work.
Sharing results
The results are shared with Riverside Dental in plain words: we can now tell within half an hour whether a new advisory affects you, and the remaining items have dates. The practice manager says this is the first time a provider has shown them evidence rather than assurances.
Priya also adds a note to the measurement plan to repeat the comparison every quarter, so improvement or slippage is visible over time.
Honest framing
She avoids claiming the changes made clients safe; she claims measured improvements in specific areas.
Cost
The checks cost about two staff days to build and run cheaply.
Worked example
Kwame asks Priya to present the results to the Cobalt team. She shows counts first, then percentages, then limits, and ends with remaining items.
Synthetic case. Figures are synthetic.
Errors and limits
Before-and-after comparisons do not prove cause; describe other changes.
Exercises
- EX067 — Lab results. Give the three before-and-after figures. Rubric: 3 = 3→1, 7→2, 6h→0.5h; 2 = two; 1 = one; 0 = none.
- EX068 — Fair test. How was the advisory time measured fairly? Rubric: 3 = unannounced synthetic advisory; 2 = partial; 1 = vague; 0 = none.
- EX069 — Limits. What limits did Priya state? Rubric: 3 = small counts, one period, other changes; 2 = two; 1 = one; 0 = none.
Synthesis
Measure again, compare fairly, and state limits.
Continue studying — chapter 23 of 30
← Chapter 22 (draft) · Course chapter index · Chapter 24 (draft) →