Chapter 19 — Critical infrastructure and sector security: monitoring
Lesson map
Lesson 19 published. Map, example, EX055.
flowchart TD
N0["Learning outcomes and connection"]
N1["19.1 Where rules come from"]
N2["19.2 Absence is a signal"]
N3["19.3 Events, episodes and alert quality"]
N4["19.4 Routing and first action"]
N5["19.5 Monitoring the monitoring"]
N6["19.7 Practical activity: rules over events"]
N7["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX055"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["19.1 Where rules come from"]
N2["19.2 Absence is a signal"]
N3["19.3 Events, episodes and alert quality"]
N4["19.4 Routing and first action"]
N5["19.5 Monitoring the monitoring"]
N6["19.7 Practical activity: rules over events"]
N7["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX055"]
[Learning outcomes and connection]
|
v
[19.1 Where rules come from]
|
v
[19.2 Absence is a signal]
|
v
[19.3 Events, episodes and alert quality]
|
v
[19.4 Routing and first action]
|
v
[19.5 Monitoring the monitoring]
|
v
[19.7 Practical activity: rules over events]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX055]
Learning outcomes and connection
After this chapter you should be able to:
- write detection rules from scenarios and from the declared architecture;
- detect absence — missing heartbeats, silent sources — as well as presence;
- tune rules to alert once per episode rather than per event, without losing information;
- monitor the monitoring: know when a rule, a source or a forwarder has stopped working.
Chapter 17 produced normalized logs. This chapter turns them into alerts that a person on call can act on. The aim is not many alerts; it is the right few, each pointing to a scenario and a first action.
19.1 Where rules come from
Three sources produce LANTERN's rules. Scenarios (Chapter 13): each step on a path that leaves a trace is a candidate rule — vendor login outside an approved window, chlorine outside its band. Architecture (Chapters 2 and 8): any flow that violates a declared conduit — corporate to station on an industrial port — is a rule that needs no threat intelligence, because it is defined by your own design. Known behaviours: repeated failed logins from one source are worth knowing about. MITRE's ATT&CK for ICS matrix catalogues adversary behaviours observed against industrial systems and helps check whether rules cover the techniques most relevant to your scenarios (MITRE ATT&CK for ICS).
19.2 Absence is a signal
In critical services, many of the most important alerts are about things that stop happening: a station that stops reporting, a relay that stops forwarding, a log source that falls silent, a backup that does not run. Absence rules need an expected rhythm (a heartbeat every five minutes) and a tolerance (alert after 15 minutes of silence). They catch outages, failed devices, cut cables, misconfigurations — and attackers who disable logging.
Rule C flagged PS-09's 40-minute silence at 23:00, 15 minutes after its last heartbeat; that 15-minute tolerance is the design choice.
19.3 Events, episodes and alert quality
A rule that fires for every matching event floods the on-call technologist: 14 failed logins become 14 alerts. Better rules alert once per episode — the first event in a window, with a count of the rest — and attach context: source zone, asset criticality, related recent events. Track each rule's false-alarm rate and its response time. A rule that has fired a hundred times without a true positive is training people to ignore it; tune it, change its threshold, or route it to a daily report instead of a page.
19.4 Routing and first action
Every alert needs an owner and a first action written in the rule: "Corporate→station Modbus: check whether an approved engineering session is in progress (change calendar); if not, call Water control room, then preserve flow records." Alerts that affect the physical process go to Water operations as well as FSTS. Write first actions before turning a rule on.
Rule B's first action did its job on the worked night: change calendar first, then the Water control room.
Rule C's first action, after the Chapter 24 study, is "call the Water control room, then check carrier status".
19.5 Monitoring the monitoring
Rules, sources and forwarders fail silently. Monitor: each source's event rate (an absence rule for logs); the normalizer's unparsed count (Chapter 17); each rule's last evaluation time; and periodic test events that should produce known alerts (Chapter 23). NIST's guidance on information security continuous monitoring emphasizes ongoing awareness of whether controls remain effective (NIST SP 800-137).
Lab 23's Rule B misses, flows from a station switch never exported to the SIEM, show the cost of not monitoring sources: the rule was right, its input was missing.
19.6 Worked example: one night, three rules (synthetic)
Lab 19 applies three rules to a night of events and heartbeats. Rule A (≥10 failed logins from one source in 5 minutes) fires once for 192.0.2.44. Rule B (corporate→station Modbus) fires twice for the same workstation, at 22:16 and 22:31 — two pages for one episode; it should be tuned to alert once and update a count. Rule C (heartbeat missing > 15 minutes) fires at 23:00 for PS-09, which was silent for 40 minutes. Note what does not fire: a single failed login by e.strand from an internal address, below Rule A's threshold — correctly, since one mistyped password is not an episode.
Taken with Chapter 17's timeline, Rule B is the alert that matters most: the workstation that logged in as an administrator at 22:14 then talked Modbus to a PLC. First action: check the change calendar; if no engineering session was approved, call the Water control room.
19.7 Practical activity: rules over events
Lab 19 (save as lab19.py, run with python3 lab19.py):
# Lab 19 - three monitoring rules over normalized events, including "absence" (offline, standard library only)
# Synthetic events for one night at Fallowmere (times are minutes after 22:00 UTC).
events = [ # minute, host, type, fields
(14, "rly-01", "auth_ok", {"src": "10.40.3.7", "user": "b.lind"}),
*[(15 + i // 4, "rly-01", "auth_fail", {"src": "192.0.2.44", "user": "root"}) for i in range(14)],
(16, "fw-hub", "flow", {"src": "10.40.3.7", "dst": "10.20.0.11", "dport": 502}),
(31, "fw-hub", "flow", {"src": "10.40.3.7", "dst": "10.20.0.11", "dport": 502}),
(44, "rly-01", "auth_fail", {"src": "10.40.9.9", "user": "e.strand"}),
]
heartbeats = {"PS-04": list(range(0, 120, 5)), "PS-07": list(range(0, 120, 5)),
"PS-09": [m for m in range(0, 120, 5) if not 50 <= m < 85]}
alerts = []
# Rule A: >=10 failed logins from one source within 5 minutes
fails = [(m, f["src"]) for m, h, t, f in events if t == "auth_fail"]
for src in {s for _, s in fails}:
times = sorted(m for m, s in fails if s == src)
for i in range(len(times)):
if sum(1 for t in times if times[i] <= t < times[i] + 5) >= 10:
alerts.append((times[i], "A brute-force", src)); break
# Rule B: any flow from corp to a station on an industrial port (not a declared conduit)
for m, h, t, f in events:
if t == "flow" and f["src"].startswith("10.40.") and f["dst"].startswith("10.20.") and f["dport"] == 502:
alerts.append((m, "B corp->station Modbus", f"{f['src']}->{f['dst']}"))
# Rule C: heartbeat missing for > 15 minutes (silence is a signal)
for st, beats in heartbeats.items():
for a, b in zip(beats, beats[1:]):
if b - a > 15:
alerts.append((a + 15, "C station silent", f"{st} gap {b - a} min"))
for m, rule, detail in sorted(alerts):
print(f"22:{m:02d}+ {rule:<24} {detail}" if m < 60 else f"23:{m-60:02d}+ {rule:<24} {detail}")
print("alerts:", len(alerts), "| rule B fired per event, not per episode:", sum(1 for a in alerts if a[1].startswith("B")))
Expected output (Python 3.13.5):
22:15+ A brute-force 192.0.2.44
22:16+ B corp->station Modbus 10.40.3.7->10.20.0.11
22:31+ B corp->station Modbus 10.40.3.7->10.20.0.11
23:00+ C station silent PS-09 gap 40 min
alerts: 4 | rule B fired per event, not per episode: 2
Count the alerts by episode, not by line: the output's four alerts are three episodes (Rule A for 192.0.2.44, Rule B for workstation 10.40.3.7, Rule C for PS-09), and tuning Rule B to alert once and update a count brings it to three. Check what stays silent too: the single failed login by e.strand from 10.40.9.9 is below Rule A's threshold, correctly, since one mistyped password is not an episode.
19.8 Common errors and limits
- Rules with no scenario, owner or first action.
- Only presence rules; nothing notices silence.
- One alert per event, flooding responders.
- Thresholds never revisited.
- Assuming a rule works because it was deployed; it must be tested (Chapter 23).
- Sending process-relevant alerts only to IT.
Rules detect what was anticipated. Unanticipated behaviour needs people who look at data — threat hunting, reviews — and absence rules, which catch effects without knowing causes.
Rule B's two pages for one episode on the worked night are the third error.
Exercises — chapter 19
EX055 — Episode rule. Objective: reduce alert noise without losing information. Inputs: Lab 19. Constraints: standard library only. Steps: (1) change Rule B so it raises one alert per source–destination pair per 60 minutes, recording the count of matching events; (2) run and show the output; (3) explain what information is preserved. Expected evidence: code, output, explanation. Rubric: 3 = one Rule B alert with count 2 and first/last times; explanation that counts and times are preserved for investigation while the page fires once; 2 = works, count missing; 1 = not run; 0 = none.
EX056 — First actions. Objective: write actionable alerts. Inputs: Rules A, B and C. Constraints: at most three steps each; must include whom to call. Steps: write the first action for each. Expected evidence: three short procedures. Rubric: 3 = A: confirm exposure of SSH, check for any success from that source, block at edge under standard change; B: check change calendar, call Water control room if unapproved, preserve flows; C: call Water control room to confirm station state, check carrier status, check relay logs for MAC failures; 2 = one weak; 1 = generic; 0 = none.
EX057 — Monitor the monitor. Objective: design self-monitoring. Inputs: LANTERN's monitoring pipeline: sources → forwarder → normalizer → rules → paging. Constraints: one check per stage. Steps: list the check, its threshold and who is alerted. Expected evidence: table. Rubric: 3 = source event-rate absence, forwarder queue/backlog, normalizer unparsed rate, rule last-run time, paging test (scheduled synthetic page acknowledged); thresholds and recipients sensible; 2 = four stages; 1 = generic; 0 = none.
Chapter synthesis
Derive rules from scenarios, from your own architecture and from known behaviours; give each an owner and a first action. Detect absence as well as presence. Alert per episode with counts and context, track false alarms and response times, and route process-relevant alerts to operations. Monitor the monitoring, because every stage of it can fail silently.
Continue studying — chapter 19 of 30
← Chapter 18 · Course chapter index · Chapter 20 →
Continue to the next authored lesson.