Chapter 14 — Critical infrastructure and sector security: automation

Chapter 14 — Critical infrastructure and sector security: automation

Lesson map

Lesson 14 published. Map, example, EX053.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["14.1 Why hashes"]
  N2["14.2 Four outcomes, four actions"]
  N3["14.4 Safe automation in control environments"]
  N4["14.5 Protecting the automation"]
  N5["14.6 Practical activity"]
  N6["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX053"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["14.1 Why hashes"]
  N2["14.2 Four outcomes, four actions"]
  N3["14.4 Safe automation in control environments"]
  N4["14.5 Protecting the automation"]
  N5["14.6 Practical activity"]
  N6["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX053"]

[Learning outcomes and connection]
    |
    v
[14.1 Why hashes]
    |
    v
[14.2 Four outcomes, four actions]
    |
    v
[14.4 Safe automation in control environments]
    |
    v
[14.5 Protecting the automation]
    |
    v
[14.6 Practical activity]
    |
    v
[Chapter synthesis]
    |
    v
[Worked example] --> [Exercise EX053]

Learning outcomes and connection

After this chapter you should be able to:

• automate configuration drift detection with cryptographic hashes and an approved manifest;
• classify differences as missing, new, approved change or unapproved change, and act differently on each;
• design automation that is safe in control environments: read-only by default, small blast radius, preserve-before-revert, and human approval for anything that touches the process;
• protect the automation itself, since a tool that can change every host is a target.

Chapters 10–13 changed, verified and troubleshot by hand. This chapter makes the repeatable checks automatic, so that a technician's attention goes to the differences that matter.

14.1 Why hashes

A SHA-256 digest (FIPS 180-4) of a file changes if any byte changes, and it is infeasible to construct a different file with the same digest. Comparing digests against a manifest recorded at a known-good moment tells you which files changed, cheaply and without storing copies. It does not tell you how they changed — for that you need the approved copy, kept in version control — and it does not tell you whether the change was legitimate — for that you need the change records. Python's hashlib provides SHA-256 in the standard library (Python hashlib).

14.2 Four outcomes, four actions

The most dangerous automation mistake is "revert all drift automatically". It destroys the evidence of an unapproved change and may undo an emergency fix made at 03:00 that has not yet been written up. Automated detection is cheap and safe; automated correction needs a short, explicit list of what may be auto-fixed, and nothing that touches process control.

14.3 Worked example: drift on hist-02 (synthetic)

A nightly job compares hist-02's key files with the manifest approved after CHG-2207. Lab 14 simulates one night, using a temporary directory.

Lab 14 (save as lab14.py, run with python3 lab14.py):

# Lab 14 - detect configuration drift against an approved manifest (offline, standard library only)
# Files are created in a temporary directory that is removed at the end.
import hashlib, json, pathlib, tempfile
def sha(p): return hashlib.sha256(p.read_bytes()).hexdigest()
with tempfile.TemporaryDirectory() as tmp:
    root = pathlib.Path(tmp)
    files = {"sshd_config": "ListenAddress 10.90.0.21\nPermitRootLogin no\nPasswordAuthentication no\n",
             "ntp.conf": "server 10.90.0.5 iburst\n",
             "fw-01.rules": "R1 allow 10.20.0.10 10.10.0.0/16 tcp 502\nR7 deny any any\n"}
    for n, t in files.items(): (root / n).write_text(t)
    manifest = {n: sha(root / n) for n in files}                       # approved baseline
    approved_changes = {"ntp.conf": "CHG-2210"}
    (root / "ntp.conf").write_text("server 10.90.0.5 iburst\nserver 10.90.0.6 iburst\n")   # approved
    (root / "sshd_config").write_text(files["sshd_config"].replace("PermitRootLogin no", "PermitRootLogin yes"))
    (root / "debug.sh").write_text("nc -l -p 9000\n")                  # new, unexpected file
    current = {p.name: sha(p) for p in sorted(root.iterdir())}
    for name in sorted(set(manifest) | set(current)):
        if name not in current: print(f"MISSING   {name}")
        elif name not in manifest: print(f"NEW       {name}  -> investigate before removing")
        elif current[name] != manifest[name]:
            tag = f"approved ({approved_changes[name]}) -> update baseline" if name in approved_changes else "UNAPPROVED -> preserve, then revert"
            print(f"CHANGED   {name}  {tag}")
        else: print(f"unchanged {name}")
    print("manifest digest", hashlib.sha256(json.dumps(manifest, sort_keys=True).encode()).hexdigest()[:16])

Expected output (Python 3.13.5):

NEW       debug.sh  -> investigate before removing
unchanged fw-01.rules
CHANGED   ntp.conf  approved (CHG-2210) -> update baseline
CHANGED   sshd_config  UNAPPROVED -> preserve, then revert
manifest digest fdea74f1d09e359c

fw-01.rules is unchanged. ntp.conf changed and CHG-2210 covers it (a second NTP server was added): verify that time is synchronized, then update the manifest. sshd_config changed with no record — someone re-enabled root login. debug.sh is new and contains a listener command. Both are preserved (copy, hash, timestamps, the account that last wrote them if the host records it) before anything is reverted, and the second is very likely to become an incident. The manifest's own digest is printed so that the manifest itself can be checked; store that digest somewhere the host's administrators cannot change (Chapter 18).

14.4 Safe automation in control environments

• Read-only first. Most valuable automation only collects and compares: configurations, firmware versions, account lists, firewall rules, certificate expiry.
• Small blast radius. Act on one host or a few; halt if many hosts differ in the same way, which usually means the desired state or the collector is wrong.
• Rate and timing. Collection traffic to field devices must respect the poll budget (Chapter 1) and never run during process-critical periods without agreement.
• Human approval for every change that could affect the process, however small.
• Log every run to the collector, with what was read and what was changed.

14.5 Protecting the automation

The automation account can read or change every host it manages, so: keep its credentials in a vault and rotate them; restrict where it may connect from; require review of changes to its desired state; alert on runs outside the schedule. The CISA advisory on long-term access to critical infrastructure describes actors using legitimate administration tools and valid accounts (CISA AA24-038A); an over-privileged automation account is exactly such a tool.

A tool that can change every host is as attractive to an adversary as an engineering workstation. Review changes to its desired state the same way you review firewall changes.

14.6 Practical activity

Extend Lab 14 so that, for an unapproved change, it writes a preservation record — file name, old digest, new digest, size and modification time — into the temporary directory before printing the finding, and prints the preservation record's own SHA-256.

Write the preservation record before printing the finding, and include the record's own digest in the output. That way a later investigator can show that the copy taken at detection time matches what the job claimed to save. Keep the temporary directory's cleanup after the hashes are recorded.

14.7 Common errors and limits

• Hashing only; no approved copy to compare content with.

• Lab 14 ignores permissions, ownership and timestamps, which matter as much as content (a world-writable service file is drift).

• Automatically reverting all drift and destroying evidence of an unapproved change.

• Storing the manifest on the host it protects.
• Hashing content only and ignoring permissions that make a service file world-writable.
• Polling field devices so aggressively that the poll cycle fails.
• Reporting drift without recording which approved baseline version it was compared against.

Exercises — chapter 14

EX053 — Classify and act. Objective: apply the four outcomes. Inputs: Lab 14 output. Steps: write the action and owner for each line, in order of execution. Expected evidence: ordered list. Rubric: 3 = preserve sshd_config and debug.sh first (copy, hash, metadata), notify security lead; investigate debug.sh before removal; revert sshd_config under a change after preservation; verify NTP and update manifest for ntp.conf; nothing for fw-01.rules; 2 = order wrong; 1 = revert all; 0 = none.

EX054 — Auto-fix list. Objective: limit automation. Inputs: candidate auto-fixes: re-enable log forwarding, reset sshd_config, restart historian service, correct NTP server list, push firewall rules, re-set PLC setpoint. Steps: mark each as auto-fix, ticket, or never-automate, with reasons. Expected evidence: six rows. Rubric: 3 = log forwarding and NTP list auto-fix (restore visibility and time, low process risk); sshd_config and firewall push ticketed (preservation and lock-out risk); historian restart ticketed with operations; PLC setpoint never automated by security tooling; 2 = one wrong; 1 = all auto; 0 = none.

EX055 — Halt guard. Objective: reduce blast radius. Inputs: a fleet of 14 hosts; on one night 9 show ntp.conf changed identically. Steps: propose a guard condition and explain what the pattern likely means. Expected evidence: guard and explanation. Rubric: 3 = halt when more than a set share (for example 25 %) of hosts show the same change; most likely an unrecorded but deliberate fleet change or a wrong manifest — confirm with the change calendar before any action; still treat as a finding until explained; 2 = guard without interpretation; 1 = revert all nine; 0 = none.

EX056 — Preservation record. Objective: extend the lab. Inputs: activity 14.6. Steps: implement and show output. Expected evidence: code and output. Rubric: 3 = record written before the finding line, contains the required fields, its digest printed; explains why the record's digest must be stored off-host; 2 = record without digest; 1 = attempted; 0 = none.

Chapter synthesis

Automate the checks that never change: collect, hash, compare with an approved manifest and classify differences. Act differently on approved, unapproved, new and missing items — preserve before you revert, and keep automatic correction to a short list that never touches the process. Protect the automation as the powerful tool it is.

Continue studying — chapter 14 of 18

← Chapter 13 · Course chapter index · Chapter 15 →

Continue to the next authored lesson.