Chapter 27 — Cloud and container security: verification
Lesson map
Lesson 27 published. Map, example, EX079.
flowchart TD
N0["Learning outcomes"]
N1["27.1 Why verify"]
N2["Lab — Lab — control verification"]
N3["27.2 Lab result"]
N4["27.3 Investigating"]
N5["27.4 Safe tests"]
N6["27.5 Evidence"]
N7["27.6 Regular runs"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX079"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["27.1 Why verify"]
N2["Lab — Lab — control verification"]
N3["27.2 Lab result"]
N4["27.3 Investigating"]
N5["27.4 Safe tests"]
N6["27.5 Evidence"]
N7["27.6 Regular runs"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX079"]
[Learning outcomes]
|
v
[27.1 Why verify]
|
v
[Lab — Lab — control verification]
|
v
[27.2 Lab result]
|
v
[27.3 Investigating]
|
v
[27.4 Safe tests]
|
v
[27.5 Evidence]
|
v
[27.6 Regular runs]
|
v
[Worked example] --> [Exercise EX079]
Learning outcomes
You will (1) verify that controls behave as intended with safe tests; (2) compare expected and observed results; (3) investigate failures; (4) record evidence.
27.1 Why verify
Configuration can say a control is on while it does not work, for example because of a wrong label or an old version. Verification observes behaviour.
The failed lab test is an example: the Restricted profile was configured for the cluster, but the test namespace lacked the label that applied it, so a privileged pod was admitted. Reading the configuration would have suggested the control worked; trying to break it showed it did not. Samuel's lab test list covers the main controls.
Lab — Lab — control verification
"""Compare expected and observed results of synthetic control tests (stdlib only)."""
tests = [("anonymous pull refused", "refused", "refused"), ("public bucket blocked", "blocked", "blocked"),
("privileged pod rejected", "rejected", "admitted"), ("MFA required for console", "required", "required"),
("audit log written", "written", "written")]
fails = [t for t in tests if t[1] != t[2]]
for name, exp, obs in tests: print(f"{'PASS' if exp == obs else 'FAIL'} {name:26} expected {exp:9} observed {obs}")
print(f"{len(tests)-len(fails)}/{len(tests)} passed")
Run with python3 lab_verify.py. Verified output:
PASS anonymous pull refused expected refused observed refused
PASS public bucket blocked expected blocked observed blocked
FAIL privileged pod rejected expected rejected observed admitted
PASS MFA required for console expected required observed required
PASS audit log written expected written observed written
4/5 passed
27.2 Lab result
Four of five tests pass: anonymous pull refused, public bucket blocked, MFA required for console and audit log written. One fails: a privileged test pod was admitted when it should have been rejected.
Four passing tests confirm that the controls for the opening case's main weaknesses, registry access and audit logging, work as intended. The one failure shows why each control needs its own test: a single missing label undermined pod security in that namespace while everything else looked fine.
27.3 Investigating
The failure traces to the test namespace lacking the label that enforces the Restricted profile. The label is added and the test passes; a check now ensures every application namespace carries it.
The fix had two parts: adding the label to the namespace, and adding a check so that any application namespace without it is reported. The second part matters more, because it prevents the same mistake in namespaces created later. Samuel reran the full test set after the fix to confirm nothing else had changed.
27.4 Safe tests
Tests use synthetic pods and objects in a test namespace or account; they never touch student data.
A test that tries to create a privileged pod in a production namespace could succeed if the control is broken, which is exactly what the test is looking for. Running it in a dedicated test namespace means a failure exposes nothing, while still showing whether the cluster's controls behave as expected. Synthetic objects are named with a test prefix and deleted after each run, so they never linger or confuse other checks.
27.5 Evidence
Each test run records date, environment, expected and observed results and the configuration version.
Recording the configuration version ties each result to the exact state that was tested. If a later change breaks a control, the evidence shows when it last passed and which change came after. Auditors can see both the result and the conditions under which it was obtained. Samuel stores the results in the same evidence log as his exports, with a hash of each test output. This makes test evidence as trustworthy as the configuration data it complements, and keeps all project evidence in one place.
27.6 Regular runs
Tests run after changes and monthly.
Running after changes catches breakage close to its cause, when it is easiest to fix. Monthly runs catch drift that no single change caused, such as a provider update. Results go into the control register, which updates each control's last test date automatically. The monthly run happens on a fixed day so its results line up with the configuration checks. When a monthly run fails a test that passed after the last change, the team knows something drifted between, which narrows the search. Results older than a month mark the control as unverified in the register.
Writing good tests
A good verification test checks one behaviour, uses synthetic inputs, has a clear expected result and cleans up after itself. Tests that leave objects behind can confuse later checks. Samuel writes each test with a name that says what it proves, so a failure message is immediately understandable. He also includes one positive test for each control, checking that compliant actions still succeed, because a control that blocks everything would pass the negative tests while breaking the service.
Worked example
Running the same tests in production's dedicated test namespace, all five pass. Samuel notes that verification in one namespace does not prove every namespace is labelled correctly, which is why the label check across namespaces was added.
Synthetic case. Tests are synthetic.
Errors and limits
A passing test shows one case works; coverage of cases matters.
Exercises
- EX079 — Lab result. Which test failed? Rubric: 3 = privileged pod admitted; 2 = partial; 1 = vague; 0 = none.
- EX080 — Cause. What caused the failure? Rubric: 3 = namespace missing Restricted label; 2 = partial; 1 = vague; 0 = none.
- EX081 — Coverage point. Why add a label check across namespaces? Rubric: 3 = one namespace test doesn't prove all; 2 = partial; 1 = vague; 0 = none.
Synthesis
Observe behaviour, not just configuration.
Continue studying — chapter 27 of 40
← Chapter 26 (draft) · Course chapter index · Chapter 28 (draft) →