Chapter 40 — Verification laboratory II
Lesson map
Lesson 40 published. Map, example, EX118.
flowchart TD
N0["Learning outcomes"]
N1["40.1 Drift"]
N2["Lab — Lab — configuration drift"]
N3["40.2 Lab result"]
N4["40.3 Classification"]
N5["40.4 Response"]
N6["40.5 Evidence"]
N7["Weekly drift report"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX118"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["40.1 Drift"]
N2["Lab — Lab — configuration drift"]
N3["40.2 Lab result"]
N4["40.3 Classification"]
N5["40.4 Response"]
N6["40.5 Evidence"]
N7["Weekly drift report"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX118"]
[Learning outcomes]
|
v
[40.1 Drift]
|
v
[Lab — Lab — configuration drift]
|
v
[40.2 Lab result]
|
v
[40.3 Classification]
|
v
[40.4 Response]
|
v
[40.5 Evidence]
|
v
[Weekly drift report]
|
v
[Worked example] --> [Exercise EX118]
Learning outcomes
You will (1) verify declared versus live configuration across the estate; (2) detect drift; (3) classify and respond to drift; (4) record evidence.
40.1 Drift
Drift is any difference between the declared configuration in version control and what is live. It may be harmless (an autoscaler changed replicas) or serious (a network policy deleted).
Drift appears in ordinary ways: a hand-run command during an incident, a console change to test something, an automated controller adjusting a value it owns, or a resource recreated without its policies. The declared state is the reference because it was reviewed and approved, while the live state shows only what happened to be applied. Comparing the two regularly shows where approved intent and reality have parted, and which differences matter.
Lab — Lab — configuration drift
"""Compare desired (declared) settings with live synthetic settings and report drift (stdlib only)."""
desired = {"meterhub/replicas": 3, "meterhub/image": "meterhub@sha256:aa11", "meterhub/runAsNonRoot": True,
"billing/replicas": 2, "billing/netpol": "default-deny", "reports/limits.cpu": "500m"}
live = {"meterhub/replicas": 3, "meterhub/image": "meterhub@sha256:bb22", "meterhub/runAsNonRoot": True,
"billing/replicas": 4, "billing/netpol": None, "reports/limits.cpu": "500m"}
drift = [(k, desired[k], live.get(k)) for k in desired if live.get(k) != desired[k]]
for k, d, l in drift:
print(f"DRIFT {k:22} desired {d} live {l}")
print(f"{len(drift)}/{len(desired)} settings drifted")
Run with python3 lab_drift.py. Verified output:
DRIFT meterhub/image desired meterhub@sha256:aa11 live meterhub@sha256:bb22
DRIFT billing/replicas desired 2 live 4
DRIFT billing/netpol desired default-deny live None
3/6 settings drifted
40.2 Lab result
Three of six synthetic settings have drifted. meterhub/image runs digest bb22 instead of the declared aa11. billing/replicas is 4 instead of 2. billing/netpol is missing instead of default-deny. The missing network policy is serious and urgent; the image digest needs investigation (an unapproved image?); the replica count is likely the autoscaler and should be excluded from declared state.
Ranking the three by urgency is the lab's real exercise. The policy comes first because its absence lets any pod reach billing now, the digest second because the running code may be unreviewed, and the replica count last.
40.3 Classification
Security-relevant drift (policies, images, identities, privileges) alerts immediately; operational drift is reported weekly.
The split follows impact. A missing network policy or an unapproved image changes what an attacker could do right now, so waiting a week would leave exposure open. A replica count or a resource limit changing affects cost and performance but rarely security. The categories are written down in advance with examples, so the person classifying a new drift does not have to decide under pressure which bucket it belongs in.
40.4 Response
Serious drift is reverted by reconciliation and investigated: who changed it, why, and was anything exposed in the meantime?
Reverting first limits exposure, but the investigation still matters, because the cause may recur. If the change came through an emergency path, the path needs fixing; if someone changed settings by hand to troubleshoot, the guidance needs updating. The question of exposure is answered from logs: for the missing network policy, Elena checks flow logs for connections that the policy would have blocked during the period it was absent.
40.5 Evidence
Each drift event is recorded with timestamps, cause and resolution.
A record holds the resource, the declared and live values, when the drift was detected, when it began if that can be established, the classification, the cause and the fix. Records support audits and show trends: repeated drift in one namespace suggests that its team lacks a safe way to make a change they need. Elena reviews the records quarterly and links each recurring cause to an improvement item, so repeated drift leads to a change in the process.
Weekly drift report
The weekly report covers resources outside GitOps, such as cloud account settings, comparing them with templates in version control. It lists each difference, its classification and an owner.
Investigating drift
For each serious drift, the investigation records who changed it, through which path, and whether anything was exposed while it lasted.
Lab rerun
The lab is rerun after each fix to confirm drift is cleared.
Worked example
In the lab scenario, the image digest change traces to an emergency hotfix built correctly by the pipeline but deployed by hand; the declared state is updated through a pull request. The missing network policy traces to a namespace recreated during troubleshooting; reconciliation restores it, and a rule now prevents namespace deletion without approval.
Synthetic case. Settings and events are synthetic.
Errors and limits
Fields that legitimately change, like replica counts, must be excluded or drift reports become noise.
Exercises
- EX118 — Lab drift. What three drifts did the lab find? Rubric: 3 = image digest, replicas, missing netpol; 2 = two; 1 = one; 0 = none.
- EX119 — Replicas. Why is the replica drift likely harmless? Rubric: 3 = autoscaler; 2 = partial; 1 = vague; 0 = none.
- EX120 — Netpol cause. What caused the missing policy? Rubric: 3 = namespace recreated during troubleshooting; 2 = partial; 1 = vague; 0 = none.
Synthesis
Detect, classify and investigate drift from declared state.
Continue studying — chapter 40 of 50
← Chapter 39 (draft) · Course chapter index · Chapter 41 (draft) →