◇ OpenCyber Academy
Engineering · Chapter 32 · English

Chapter 32 — OT, IoT and cyber-physical security: reliability

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 32 — OT, IoT and cyber-physical security: reliability

Lesson map

Lesson 32 published. Map, example, EX094.

flowchart TD
  N0["Learning outcomes"]
  N1["32.1 Targets"]
  N2["32.2 Retries"]
  N3["32.3 Safe degradation"]
  N4["Practice note"]
  N5["32.1b Error budgets and health checks"]
  N6["Field note — queue back-pressure"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX094"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["32.1 Targets"]
  N2["32.2 Retries"]
  N3["32.3 Safe degradation"]
  N4["Practice note"]
  N5["32.1b Error budgets and health checks"]
  N6["Field note — queue back-pressure"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX094"]

[Learning outcomes]
    |
    v
[32.1 Targets]
    |
    v
[32.2 Retries]
    |
    v
[32.3 Safe degradation]
    |
    v
[Practice note]
    |
    v
[32.1b Error budgets and health checks]
    |
    v
[Field note — queue back-pressure]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX094]

Learning outcomes

You will (1) set availability targets for OT security tools; (2) use retries with budgets that respect cycle time; (3) degrade safely when a dependency fails.

32.1 Targets

Allow/deny path during windows: high availability. Coverage dashboards: best-effort. State the target; do not imply five-nines without a budget.

At ABP, the deny path on the serial monitor must work throughout every window, while the coverage dashboard may be unavailable for an hour without harm. Lena writes both targets down with the budget behind each, so nobody assumes the dashboard needs the same care as enforcement, or that enforcement can tolerate the dashboard's outages. Targets are reviewed each year.

32.2 Retries

Retries help flaky uploads of evidence but need a deadline so the trunk budget still holds. Cap total attempts.

Lena's evidence uploader at ABP retries with increasing delays and gives up after a fixed number of attempts or a deadline, whichever comes first. Failed uploads stay queued on disk and are reported, so nothing is lost. Retries never run on the trunk to PLC-A, where they could crowd out control traffic and break the cycle budget. The limits are written in the permit.

32.3 Safe degradation

If the finding store is down, continue deny/allow locally and queue drafts on disk — never fail open on policy checks.

Synthetic case — Ashford Batch Plant. During a findings-store outage, Lena's laptop queue held drafts; the serial monitor kept denying out-of-range writes.

Lena tests this degraded mode in Lab II on the bench. During a real outage of the finding store, drafts queued on Lena's laptop and the serial monitor kept denying out-of-range writes. After each outage, Lena confirms that every queued draft reached the store by comparing counts with the queue log.

32.4 Errors and limits

Retries against a rate-limited trunk can look like floods; coordinate budgets with the work permit.

Lena's work permit at ABP states the maximum request rate for any assessment traffic on each trunk, including retries. If retries would exceed it, the tool waits instead. She coordinates with the site's monitoring team before a window, so a burst of retries is not mistaken for an attack and does not trigger an unnecessary incident response. Rates are set with the control engineers. They are reviewed yearly.

Practice note

When the monitor's draft-finding queue grows past a threshold, the window lead stops starting new discovery methods and is paged. Untracked scans while findings cannot be recorded create risk.

32.1b Error budgets and health checks

The monitor's deny path gets its own error budget, separate from the best-effort budget of the CNHP dashboard. Health-check policy storage and event sink at start of window. Starting a window blind and discovering a broken sink mid-scan is an operations failure this chapter is meant to prevent. A failed health check delays the window start; the checks are never shortened to save time.

Chaos experiments run only on synthetic ranges, never during a client's signed window. Record experiment ids separately from window ids so metrics do not mix. Both budgets are published.

Field note — queue back-pressure

When the draft queue hit 50 during a store outage, Track-style discovery stopped automatically and paged Lena. Deny/allow continued. The threshold of 50 was chosen so the queue holds a full night of drafts with room to spare.

32.5 Worked example — degrading safely when the collector fails (synthetic ABP)

CNHP's monitoring collector sits on a mirror port. If it fails, nothing in the process should change: the design keeps the collector out of every control path. Lena's reliability targets are about evidence, not production: 99.5 % capture availability per month and alerts on loss above 0.1 %. Retries to the central store use exponential back-off with a cap so that a store outage does not flood the DMZ link shared with the historian replica. When the queue fills, the collector drops the oldest raw packets but keeps the summary records, and it raises a "visibility degraded" alert so analysts know their absence of alerts means less.

Exercises

Synthesis

Reliable denies, budgeted retries, degrade closed.

Continue studying — chapter 32 of 50

← Chapter 31 (draft) · Course chapter index · Chapter 33 (draft) →