Chapter 4 — Baseline replication and pilot
Lesson map
Lesson 04 published. Map, example, EX021.
flowchart TD
N0["Learning outcomes"]
N1["4.1 Why replicate first"]
N2["Lab — workload checks"]
N3["4.2 Lab result"]
N4["4.3 The pilot"]
N5["4.4 Protocol changes"]
N6["4.5 CIS and provider benchmarks"]
N7["Recording the replication"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX021"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["4.1 Why replicate first"]
N2["Lab — workload checks"]
N3["4.2 Lab result"]
N4["4.3 The pilot"]
N5["4.4 Protocol changes"]
N6["4.5 CIS and provider benchmarks"]
N7["Recording the replication"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX021"]
[Learning outcomes]
|
v
[4.1 Why replicate first]
|
v
[Lab — workload checks]
|
v
[4.2 Lab result]
|
v
[4.3 The pilot]
|
v
[4.4 Protocol changes]
|
v
[4.5 CIS and provider benchmarks]
|
v
[Recording the replication]
|
v
[Worked example] --> [Exercise EX021]
Learning outcomes
You will (1) replicate the existing daily scan on synthetic workloads; (2) confirm the checks behave as specified; (3) pilot data collection; (4) adjust the protocol.
4.1 Why replicate first
Before comparing designs, Amara checks that the measurement tool reproduces what the existing scanner reports. If the checks disagree with the scanner, any later difference could come from the tool, not the control.
Agreement on the same inputs is the test. If the tool flagged fewer violations than the scanner, exposure under scanning would be undercounted; if more, the scanning arm would look worse. Either way, the later comparison would measure a difference between tools rather than between controls. Agreement on the five synthetic pods, seven violations from each, is the precondition for comparing designs at all.
Lab — workload checks
"""Check synthetic pod specs against selected Pod Security Standards checks (Baseline and Restricted profiles) (stdlib only)."""
pods = {
"billing-api": {"privileged": False, "hostPath": False, "runAsNonRoot": True, "allowPrivilegeEscalation": False},
"report-job": {"privileged": False, "hostPath": True, "runAsNonRoot": True, "allowPrivilegeEscalation": False},
"log-agent": {"privileged": True, "hostPath": True, "runAsNonRoot": False, "allowPrivilegeEscalation": True},
"web-frontend": {"privileged": False, "hostPath": False, "runAsNonRoot": False, "allowPrivilegeEscalation": False},
"scheduler": {"privileged": False, "hostPath": False, "runAsNonRoot": True, "allowPrivilegeEscalation": True},
}
total = 0
for name, s in pods.items():
v = []
if s["privileged"]: v.append("privileged")
if s["hostPath"]: v.append("hostPath volume")
if not s["runAsNonRoot"]: v.append("runAsNonRoot not set")
if s["allowPrivilegeEscalation"]: v.append("privilege escalation allowed")
total += len(v)
print(f"{name:13} {len(v)} {', '.join(v) or 'compliant'}")
print(f"violations {total}; compliant pods {sum(1 for s in pods.values() if not any([s['privileged'], s['hostPath'], not s['runAsNonRoot'], s['allowPrivilegeEscalation']]))}/{len(pods)}")
Run with python3 lab_baseline.py. Verified output:
billing-api 0 compliant
report-job 1 hostPath volume
log-agent 4 privileged, hostPath volume, runAsNonRoot not set, privilege escalation allowed
web-frontend 1 runAsNonRoot not set
scheduler 1 privilege escalation allowed
violations 7; compliant pods 1/5
4.2 Lab result
Five synthetic pod specifications are checked. billing-api is compliant. report-job has one violation (host path volume), web-frontend one (runAsNonRoot not set) and scheduler one (privilege escalation allowed). log-agent has four: privileged, host path volume, runAsNonRoot not set and privilege escalation allowed. In total seven violations; one of five pods is compliant. The existing scanner's synthetic report gives the same seven.
Matching the scanner's count is necessary but not sufficient: the same total could hide different findings on different pods. Amara compares findings pod by pod and check by check, so log-agent's four violations must match the scanner's four, and the three single violations must match in kind as well as number.
4.3 The pilot
For two weeks on two non-production clusters, Amara records each risky configuration's start time from the audit log and its end time from removal or block. The pilot tests whether timestamps are complete and whether the measurement script handles restarts and redeployments.
The two clusters are non-production on purpose: a measurement script that misread timestamps there could not affect patient services. The pilot found more than script bugs. One cluster kept audit logs for only 24 hours, too short to measure long exposures, and Meridian's platform team extended retention to 30 days for the study clusters, with the change recorded.
4.4 Protocol changes
The pilot shows that redeploying the same workload creates new pods, which would be counted as new exposures. The protocol now groups pods by owning workload, so exposure is measured per workload configuration.
Without this change, a workload with a privileged container that was redeployed three times in a day would appear as four short exposures rather than one long one, making daily scanning look better than it is. Grouping by owning workload, such as the Deployment or DaemonSet, measures what matters: how long a risky configuration of that workload was live.
4.5 CIS and provider benchmarks
The CIS Benchmarks for Kubernetes include similar checks; Amara notes overlaps so results can be related to them.
The overlap is close for the four checks in Amara's definition: privileged containers, host path volumes, running as root and privilege escalation all appear in the benchmarks' pod security recommendations. Noting the matching benchmark entries lets a reader who uses CIS rather than Pod Security Standards relate the results to their own controls without reinterpreting the definition. Her appendix places the matching benchmark recommendation beside each of the four Pod Security Standards checks: privileged containers, host path volumes, running as root and privilege escalation.
Recording the replication
The replication record lists both outputs side by side, with the tool version and input hashes.
Worked example
The pilot also reveals that one cluster's audit logs were retained for only 24 hours, too short to measure long exposures. Meridian's platform team extends retention to 30 days for the study clusters, with the change recorded.
Synthetic case. Pods, logs and counts are synthetic.
Errors and limits
A pilot is for finding problems in the method, not for estimating the effect.
Exercises
- EX021 — Purpose. Why replicate the scanner first? Rubric: 3 = differences must come from control not tool; 2 = partial; 1 = vague; 0 = none.
- EX022 — Lab totals. What totals did the lab give? Rubric: 3 = 7 violations; 1/5 compliant; 2 = partial; 1 = vague; 0 = none.
- EX023 — log-agent. What violations did log-agent have? Rubric: 3 = all four; 2 = one; 1 = vague; 0 = none.
- EX024 — Pod grouping. What protocol change followed the pilot? Rubric: 3 = group pods by owning workload; 2 = partial; 1 = vague; 0 = none.
- EX025 — Retention. What logging problem was found? Rubric: 3 = 24h retention too short; 2 = partial; 1 = vague; 0 = none.
- EX026 — Pilot purpose. What is a pilot for? Rubric: 3 = finding method problems; 2 = partial; 1 = vague; 0 = none.
Synthesis
Replicate the baseline and pilot the method before the main study.
Continue studying — chapter 04 of 08
← Chapter 03 (draft) · Course chapter index · Chapter 05 (draft) →