Chapter 30 — Verification laboratory I
Lesson map
Lesson 30 published. Map, example, EX088.
flowchart TD
N0["Learning outcomes"]
N1["30.1 Lab setup"]
N2["30.2 Admission tests"]
N3["30.3 Network tests"]
N4["30.4 Identity tests"]
N5["30.5 Recording"]
N6["30.6 Failures"]
N7["Parity checks"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX088"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["30.1 Lab setup"]
N2["30.2 Admission tests"]
N3["30.3 Network tests"]
N4["30.4 Identity tests"]
N5["30.5 Recording"]
N6["30.6 Failures"]
N7["Parity checks"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX088"]
[Learning outcomes]
|
v
[30.1 Lab setup]
|
v
[30.2 Admission tests]
|
v
[30.3 Network tests]
|
v
[30.4 Identity tests]
|
v
[30.5 Recording]
|
v
[30.6 Failures]
|
v
[Parity checks]
|
v
[Worked example] --> [Exercise EX088]
Learning outcomes
You will (1) verify controls with safe, synthetic tests; (2) check admission, network and identity controls behave as specified; (3) record evidence; (4) handle failures.
30.1 Lab setup
A test cluster mirrors production policies. Tests use synthetic workloads designed to violate one rule each, so a correct control rejects them. No production systems are touched.
Mirroring production policies ensures tests reflect what production actually enforces. Synthetic workloads designed to break exactly one rule make results unambiguous: if one is admitted, the corresponding control has failed. Keeping tests in a separate cluster means no risk to production services or data. The test cluster is rebuilt from code before each full run, so earlier experiments cannot leave state that changes results. Its version matches production, which Elena checks at the start of every run.
30.2 Admission tests
A synthetic pod requesting privileged mode must be rejected; one without runAsNonRoot must be rejected; an unsigned image must be rejected; a compliant pod must be admitted.
Each rejection test proves a specific control works; the compliant pod proves the rules are not so strict that nothing can run. Running all four together catches both failure modes: a control that allows too much, and a configuration that blocks legitimate workloads. The pods use small images built for testing, holding no real application code. Each is named after the rule it breaks.
30.3 Network tests
From a test pod labelled web, a connection to db must fail and a connection to api must succeed, matching the chapter 24 design.
Testing both a denied and an allowed path confirms the policy blocks what it should without breaking normal traffic. Using labels that match production pods ensures the policies being tested are the ones that would apply. A failure in either direction is a defect. Elena also tests DNS from the web pod, since egress rules could break name resolution. Results are stored with the policy version, so a later change can be compared directly with this run.
30.4 Identity tests
A test service with workload identity can read its allowed bucket and is refused writing to another; no keys exist for the service account.
Reading the allowed bucket confirms the role works; being refused a write elsewhere confirms permissions are limited. Checking that no keys exist confirms the old credential path is closed. Together they verify R1 for the test service. The tests use a dedicated test bucket holding synthetic files only, so no readings are involved. Elena also checks the cloud audit log afterwards: the refused write should appear there with the service's workload identity, which confirms the denial came from the role rather than from a network fault.
30.5 Recording
Each test records requirement, input, expected and actual result, time and cluster version.
Records allow anyone to see what was tested, when and against which cluster version. If a later change breaks a control, comparing records shows when it last passed. Records also form part of the evidence submitted to the regulator. Elena writes results to a CSV file per run, named with the date and cluster version. Files are kept in the evidence folder and are never edited after the run; corrections appear as a new run with a note explaining what changed and why the earlier result was wrong.
30.6 Failures
A failed test is a defect with an owner, not a test to rewrite.
Rewriting a failing test to pass hides the problem it found. Treating it as a defect means someone investigates and fixes the control. The test stays as it is and is rerun after the fix, providing evidence that the problem is resolved. The defect records which requirement is at risk, so its priority follows from the requirement's priority. Until it is fixed, the requirement is reported as not met, even if every other test passes. Elena's first run finds one such defect: a namespace label that set the wrong pod security profile, so a weaker level was enforced than the requirement demands.
Parity checks
Before each verification run, a script compares policy versions, namespace labels and admission configuration between the test cluster and production, and stops the run if they differ. This ensures results describe production. The first parity check finds the test cluster one policy version behind, which is corrected before testing.
Evidence storage
Results are stored in the repository, versioned with the policies they tested, so auditors can see exactly which policy version passed which test and when.
Worked example
One test fails: a pod with allowPrivilegeEscalation unset is admitted. Investigation shows the namespace label set the profile to Baseline, not Restricted, after a copy-paste error. The label is fixed, the test passes, and a pipeline check is added for namespace labels.
Synthetic case. Tests and results are synthetic.
Errors and limits
Verification in a test cluster assumes it matches production; check configuration parity.
Exercises
- EX088 — Test design. How are admission tests designed? Rubric: 3 = synthetic workloads violating one rule each; 2 = partial; 1 = vague; 0 = none.
- EX089 — Network test. What network results are expected? Rubric: 3 = web->db fails; web->api succeeds; 2 = partial; 1 = vague; 0 = none.
- EX090 — Failure cause. Why was the escalation test admitted? Rubric: 3 = namespace label set Baseline; 2 = partial; 1 = vague; 0 = none.
Synthesis
Safe synthetic tests prove controls behave as designed.
Continue studying — chapter 30 of 50
← Chapter 29 (draft) · Course chapter index · Chapter 31 (draft) →