Chapter 22 — Critical infrastructure and sector security: automation
Lesson map
Lesson 22 published. Map, example, EX064.
flowchart TD
N0["Learning outcomes and connection"]
N1["22.1 Configuration as code"]
N2["22.2 Plan, then act"]
N3["22.3 Safe auto-fix versus ticket"]
N4["22.5 Automation is code: treat it so"]
N5["22.7 Practical activity: drift plan"]
N6["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX064"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["22.1 Configuration as code"]
N2["22.2 Plan, then act"]
N3["22.3 Safe auto-fix versus ticket"]
N4["22.5 Automation is code: treat it so"]
N5["22.7 Practical activity: drift plan"]
N6["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX064"]
[Learning outcomes and connection]
|
v
[22.1 Configuration as code]
|
v
[22.2 Plan, then act]
|
v
[22.3 Safe auto-fix versus ticket]
|
v
[22.5 Automation is code: treat it so]
|
v
[22.7 Practical activity: drift plan]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX064]
Learning outcomes and connection
After this chapter you should be able to:
- express desired configuration as code and compare it automatically with actual state;
- design automation that plans before it acts, separates safe auto-fixes from changes needing approval, and limits its own blast radius;
- recognize when the source of truth itself may be wrong, and stop;
- keep automation within change control and produce evidence from it.
Chapters 6 and 14 described baselines and integrity checks done by hand. With three relays and a growing estate, manual checking stops happening. Automation makes routine checks reliable — and makes mistakes fast and wide. This chapter is about getting the first without the second.
22.1 Configuration as code
Write the desired state of each host class as data in version control: SSH settings, listening addresses, time servers, log forwarding, package versions. A collector reads the actual state from each host (read-only), and a comparison produces drift: keys whose actual value differs from desired. NIST's configuration-management guide describes monitoring configurations against baselines as a core activity (NIST SP 800-128); NIST's secure software development framework applies similar disciplines — version control, review, protected build pipelines — to the code that implements it (NIST SP 800-218).
22.2 Plan, then act
Good automation separates plan from apply. The plan lists every intended change: host, setting, current value, desired value, and mode. The apply step executes an approved plan, not a fresh computation. This gives a person a chance to see what will happen, gives the change record something specific to approve, and produces evidence afterwards.
Lab 22's plan for RLY-02 lists two changes, password SSH and public time servers, with current and desired values.
RLY-03's debug listener appears in the plan as a ticket rather than an action, because removing it could destroy evidence of who enabled it.
22.3 Safe auto-fix versus ticket
Some drift is safe to correct automatically: settings that are low-risk, reversible and well-understood (re-disabling SSH password authentication; restoring the approved time servers; restoring log forwarding). Other drift must go through change control: anything that may change service behaviour (a package version), anything that may be evidence (an unexpected debug listener — which may be an attacker's or an engineer's, and should be preserved before removal, Chapter 20), and anything that affects the physical process. Write the auto-fix list explicitly and keep it short.
22.4 Blast-radius limits
Automation must protect against its own mistakes. Common limits: apply to one host at a time, verifying each before the next; stop on the first failure; and halt and escalate if too much drift is found — if most of the fleet appears to have drifted, the likelier explanation is that the desired state is wrong (a bad commit, a wrong branch), and auto-fixing would push the error everywhere. A large outage caused by a faulty update distributed to many systems at once — such as the July 2024 incident affecting Windows hosts running a security product, described by Microsoft as affecting about 8.5 million devices (Microsoft, 20 July 2024) — illustrates why staged rollout and halting conditions matter for any automated distribution.
22.5 Automation is code: treat it so
The automation's own code and desired-state files are high-value: whoever can change them can change every host. Protect them with review before merge, restricted write access, signed commits or protected branches, and logs of each run. The automation's credentials to read and change hosts are privileged accounts (Chapter 7).
The NTP merge in Chapter 25 is the warning: one reviewed commit changed time on every relay.
Treat the account the automation uses to reach RLY-01 to RLY-03 like vendor-kell: time-bound where possible, logged, and reviewed in the quarterly access review of Chapter 7.
22.6 Worked example: a dry run across three relays (synthetic)
Lab 22 compares desired state with three relays. RLY-01 is compliant. RLY-02 has SSH password authentication re-enabled and public time servers — both on the auto-fix list. RLY-03 has a debug listener on all interfaces, an older relay package and no log forwarding. Two of three hosts have drifted, more than half the fleet, so the guard halts the run. Here the halt is a false alarm in one sense — the source of truth is fine — but the rule is right to stop: a person confirms the desired state, then approves applying the three auto-fixes host by host, while the debug listener and package version go to tickets. The debug listener on RLY-03 is the same symptom found by the integrity check on RLY-01 in Chapter 6: twice is a pattern, and the troubleshooting method of Chapter 20 applies.
Notice too that RLY-03's missing log forwarding means its own logs about the debug listener may never have reached the SIEM — another reason to preserve local evidence before fixing.
22.7 Practical activity: drift plan
Lab 22 (save as lab22.py, run with python3 lab22.py):
# Lab 22 - automated configuration-drift check with a dry-run plan (offline, standard library only)
# Synthetic desired state (from version control) and actual state (as collected from three relay hosts).
import json
desired = {"ssh.password_auth": False, "ssh.permit_root": False, "relay.listen": "10.30.0.5:8443",
"relay.debug_listen": None, "ntp.servers": ["10.30.0.1", "10.30.0.2"], "log.forward": "siem-01:6514",
"pkg.relayd": "2.4.1"}
actual = {
"rly-01": {**desired},
"rly-02": {**desired, "ssh.password_auth": True, "ntp.servers": ["pool.example.net"]},
"rly-03": {**desired, "relay.debug_listen": "0.0.0.0:9000", "pkg.relayd": "2.3.9", "log.forward": None},
}
SAFE_AUTOFIX = {"ssh.password_auth", "ssh.permit_root", "ntp.servers", "log.forward"} # low-risk, reversible
def plan(host, state):
steps = []
for k, want in desired.items():
have = state.get(k)
if have != want:
mode = "auto" if k in SAFE_AUTOFIX else "ticket"
steps.append({"host": host, "key": k, "have": have, "want": want, "mode": mode})
return steps
all_steps = [s for h, st in actual.items() for s in plan(h, st)]
for s in all_steps:
print(f"{s['host']} {s['key']:<20} have={json.dumps(s['have']):<22} want={json.dumps(s['want']):<26} -> {s['mode']}")
print("hosts compliant:", [h for h, st in actual.items() if not plan(h, st)])
print("dry run only: %d auto-fix steps, %d need a change ticket" % (sum(s['mode'] == 'auto' for s in all_steps), sum(s['mode'] == 'ticket' for s in all_steps)))
# Guard: never auto-apply to more than one host at a time, and stop if the host count with drift exceeds half the fleet
drifted = {s['host'] for s in all_steps}
print("guard: drifted hosts %d of %d -> %s" % (len(drifted), len(actual), "HALT and escalate (possible bad source of truth)" if len(drifted) > len(actual) / 2 else "proceed one host at a time"))
Expected output (Python 3.13.5):
rly-02 ssh.password_auth have=true want=false -> auto
rly-02 ntp.servers have=["pool.example.net"] want=["10.30.0.1", "10.30.0.2"] -> auto
rly-03 relay.debug_listen have="0.0.0.0:9000" want=null -> ticket
rly-03 log.forward have=null want="siem-01:6514" -> auto
rly-03 pkg.relayd have="2.3.9" want="2.4.1" -> ticket
hosts compliant: ['rly-01']
dry run only: 3 auto-fix steps, 2 need a change ticket
guard: drifted hosts 2 of 3 -> HALT and escalate (possible bad source of truth)
Read the halt first. Two of three relays drifted, more than half the fleet, so the run stopped before applying anything. Then read the plan per host: RLY-02's password authentication and public time servers, and RLY-03's missing log forwarding, are on the auto-fix list; RLY-03's debug listener and older relay package are not, and become tickets. Before approving any fix on RLY-03, copy its local logs, because without forwarding they may be the only record of when the debug listener appeared.
22.8 Common errors and limits
- Automation that applies changes without a reviewable plan.
- Long auto-fix lists, including settings that may be evidence or change behaviour.
- Applying to every host at once.
- No halt condition for "too much drift".
- Automation code and credentials less protected than the hosts they manage.
- Treating automated compliance as proof of security; it proves only the settings it checks.
Lab 22's halt at two of three drifted relays is the fourth item done right.
An auto-fix list that included the debug listener would have removed the evidence of who set it on RLY-03.
Exercises — chapter 22
EX064 — Approve the plan. Objective: review an automation plan as a change approver. Inputs: Lab 22 output. Constraints: approve, modify or reject each step. Steps: (1) decide for each of the five steps; (2) give the order of application; (3) state the verification after each host. Expected evidence: table. Rubric: 3 = auto-fixes approved host by host (RLY-02 then RLY-03), debug listener preserved then removed under a ticket after investigation, package upgrade as a normal change with canary (Chapter 27), log forwarding restored early on RLY-03 to regain visibility; verification checks named; 2 = order or preservation missing; 1 = approve all; 0 = none.
EX065 — Tune the guard. Objective: reason about halting thresholds. Inputs: the guard halts when more than half the fleet has drift. Constraints: the fleet will grow to 12 relays. Steps: (1) say whether "more than half" is still sensible at 12; (2) propose a better guard using the kind of drift, not just the count of hosts. Expected evidence: paragraph. Rubric: 3 = recognizes that the same key drifting on many hosts suggests a bad source of truth, while different keys on different hosts suggests local changes; proposes halting when one key differs on more than a set share of hosts, or when a commit to desired state is recent and unreviewed; 2 = partial; 1 = keeps "half"; 0 = none.
EX066 — Protect the pipeline. Objective: secure the automation itself. Inputs: the desired-state repository and the automation account. Constraints: five controls. Steps: list them with evidence. Expected evidence: table. Rubric: 3 = protected branch with required review, restricted maintainers with MFA, signed commits or tags, automation account limited to the managed hosts with credentials in a vault, run logs forwarded to SIEM with an alert on runs outside schedule; evidence for each; 2 = four; 1 = generic; 0 = none.
Chapter synthesis
Express desired state as code, collect actual state read-only, and compute drift. Plan before acting; auto-fix only a short list of low-risk, reversible settings and send everything else — especially anything that may be evidence — to change control. Limit blast radius: one host at a time, stop on failure, halt when drift looks like a bad source of truth. Protect the automation as carefully as the hosts it manages.
Continue studying — chapter 22 of 30
← Chapter 21 · Course chapter index · Chapter 23 →
Continue to the next authored lesson.