Chapter 6 — Cloud and container security: controlled evaluation
Lesson map
Lesson 06 published. Map, example, EX026.
flowchart TD
N0["Learning outcomes"]
N1["6.1 The question"]
N2["6.2 Corpus"]
N3["Lab — policy evaluation"]
N4["6.3 Lab result"]
N5["6.4 Fairness of comparison"]
N6["6.5 Remaining misses"]
N7["6.6 Decision"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX026"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["6.1 The question"]
N2["6.2 Corpus"]
N3["Lab — policy evaluation"]
N4["6.3 Lab result"]
N5["6.4 Fairness of comparison"]
N6["6.5 Remaining misses"]
N7["6.6 Decision"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX026"]
[Learning outcomes]
|
v
[6.1 The question]
|
v
[6.2 Corpus]
|
v
[Lab — policy evaluation]
|
v
[6.3 Lab result]
|
v
[6.4 Fairness of comparison]
|
v
[6.5 Remaining misses]
|
v
[6.6 Decision]
|
v
[Worked example] --> [Exercise EX026]
Learning outcomes
You will (1) evaluate policy sets on a labelled corpus; (2) measure recall and false rejections; (3) compare alternatives fairly; (4) decide using evidence.
6.1 The question
Before enforcing, Inês compares two policy-engine rule sets on past deployments: P1, a broad set copied from an example, and P2, tuned to Beacon's workloads. Comparing rule sets on past deployments shows how each would behave before it can block real work. An example set copied from documentation may include rules that do not fit Beacon's workloads, while a tuned set may miss risks its authors did not think of. Measuring both on the same deployments gives evidence for the choice instead of opinion.
6.2 Corpus
200 synthetic deployment manifests based on Beacon's recent changes, each labelled risky or not by two reviewers who agreed on the label guide beforehand. Labelling before running the rule sets means the labels cannot be influenced by which set caught what. The two reviewers agreed a guide that defines a risky deployment, such as one running as root or using an unapproved registry. Basing manifests on Beacon's recent changes keeps the corpus close to what developers actually deploy. The label guide and the 200 labels are kept with the evaluation, so later quarters reuse them.
Lab — policy evaluation
"""Compare two policy sets on a labelled synthetic deployment corpus (stdlib only)."""
# each item: (truly risky?, rejected by policy P1?, rejected by policy P2?)
corpus = [(True, True, True)] * 18 + [(True, False, True)] * 4 + [(True, False, False)] * 2 + \
[(False, True, False)] * 9 + [(False, True, True)] * 1 + [(False, False, False)] * 166
for i, name in ((1, "P1 broad"), (2, "P2 tuned")):
tp = sum(1 for r, *p in corpus if r and p[i-1]); fp = sum(1 for r, *p in corpus if not r and p[i-1])
fn = sum(1 for r, *p in corpus if r and not p[i-1])
print(f"{name}: caught {tp}/{tp+fn} risky (recall {tp/(tp+fn):.2f}), false rejections {fp}, precision {tp/(tp+fp):.2f}")
print(f"corpus {len(corpus)} deployments, {sum(1 for c in corpus if c[0])} risky")
Run with python3 lab_eval.py. Verified output:
P1 broad: caught 18/24 risky (recall 0.75), false rejections 10, precision 0.64
P2 tuned: caught 22/24 risky (recall 0.92), false rejections 1, precision 0.96
corpus 200 deployments, 24 risky
6.3 Lab result
Of 200 deployments, 24 are risky. P1 catches 18 (recall 0.75) with 10 false rejections (precision 0.64). P2 catches 22 (recall 0.92) with 1 false rejection (precision 0.96). P2 is better on both measures: it covers Beacon's actual risky patterns and avoids rules that misfire on Beacon's normal workloads. The precision difference matters in practice, because each false rejection costs a developer time and weakens trust in the controls. At Beacon's deployment rate, P1's misfiring rules would have caused about 40 false rejections a month, which is why precision weighs as heavily as recall here. Inês records both results with the rule versions, so the quarterly re-evaluation can compare against them.
6.4 Fairness of comparison
Both sets run on the same corpus, labels were fixed before running, and the tuning of P2 used a different sample to avoid fitting the test set. Using a separate sample for tuning prevents P2 from looking good only because it was fitted to the same deployments it was tested on. Fixing labels first stops reviewers adjusting them after seeing results. Inês records the corpus, labels, rule versions and tuning sample, so the comparison can be repeated when rules change. The quarterly re-evaluation uses the same method on newly added manifests from Beacon's recent changes.
6.5 Remaining misses
Two risky deployments pass both sets; both use a host path volume pattern not yet covered, which becomes a new rule. A miss that both sets share shows a gap in Beacon's rules rather than a weakness of one set. Adding a rule for the host path pattern closes that gap, and Inês checks the new rule against the corpus to confirm it catches both deployments without adding false rejections. She records the host path rule and its corpus result in the policy repository's change log. The new host path rule enters P2 before any cluster moves to enforce mode.
6.6 Decision
Enforce P2, add the new rule, and re-evaluate quarterly. Enforcing P2 gives Beacon the better recall and far fewer false rejections. Adding the host path rule closes the gap both sets shared. Re-evaluating each quarter keeps the rules matched to Beacon's changing workloads, because a rule set tuned today may misfire on new deployment patterns next year. Inês adds the quarterly evaluation to the security team's assurance schedule. The platform team signs off the decision, and the evaluation report goes to the risk committee with the corpus description and both rule sets' results, so the choice can be checked later.
Reporting the evaluation
The evaluation report states the question, corpus, labelling method, both results, the limitation of a historical corpus, and the decision. Prof. Rao notes that recording the inter-reviewer agreement on labels would strengthen it, so Inês adds the agreement figure from the reviewers' calibration session and the method used to settle disagreements.
Worked example
The platform team asks whether 10 false rejections from P1 really matter. Inês estimates each costs a developer about an hour and erodes trust in the controls; at Beacon's deployment rate, P1 would cause about 40 false rejections a month.
Synthetic case. Corpus and results are synthetic.
Errors and limits
A corpus from past deployments may not represent future workloads.
Exercises
- EX026 — Lab figures. Give recall and precision for P1 and P2. Rubric: 3 = P1 0.75/0.64; P2 0.92/0.96; 2 = two; 1 = one; 0 = none.
- EX027 — Fairness. How was the comparison kept fair? Rubric: 3 = same corpus, fixed labels, separate tuning sample; 2 = two; 1 = one; 0 = none.
- EX028 — Misses. What did the two misses share? Rubric: 3 = host path pattern not covered; 2 = partial; 1 = vague; 0 = none.
- EX029 — False rejection cost. How did Inês estimate false rejection cost? Rubric: 3 = hour each, ~40/month, trust; 2 = partial; 1 = vague; 0 = none.
- EX030 — Decision. What was decided? Rubric: 3 = enforce P2 plus new rule, quarterly re-evaluation; 2 = partial; 1 = vague; 0 = none.
Synthesis
Evaluate controls on labelled evidence before enforcing them.
Continue studying — chapter 06 of 12
← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →