◇ OpenCyber Academy
Bachelor's · Chapter 38 · English

Chapter 38 — Capstone implementation

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 38 — Capstone implementation

Lesson map

Lesson 38 published. Map, example, EX112.

flowchart TD
  N0["Learning outcomes"]
  N1["38.1 Increments"]
  N2["38.2 Decision records"]
  N3["38.3 Evaluation"]
  N4["38.4 Reading the confusion matrix"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX112"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["38.1 Increments"]
  N2["38.2 Decision records"]
  N3["38.3 Evaluation"]
  N4["38.4 Reading the confusion matrix"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX112"]

[Learning outcomes]
    |
    v
[38.1 Increments]
    |
    v
[38.2 Decision records]
    |
    v
[38.3 Evaluation]
    |
    v
[38.4 Reading the confusion matrix]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX112]

Learning outcomes

You will (1) implement a capstone artefact in small, tested increments; (2) maintain a change log and decision record; (3) evaluate the artefact on a labelled set with a confusion matrix; and (4) report precision and recall with their denominators.

38.1 Increments

Build the smallest end-to-end version first: read a capture file, classify each write, print decisions. Then add rules one at a time, each with tests (Chapter 8) and conformance vectors (Chapter 27). A working small tool in week 4 beats an ambitious one that never runs.

Maya's week-4 version read a synthetic capture, classified each Modbus write against the Chapter 27 allow-list and printed one decision per line. The rate rule came later, with its own tests, and the cumulative-change rule only after the evaluation showed the drift attack was missed. Each increment could be run and checked against the eight labelled cases before the next one began.

38.2 Decision records

When you choose, record why: "Decision 7: treat writes during a scheduled maintenance window as benign-maint only if the ticket ID appears in the window list; alternatives considered: time-only rule (rejected: insiders could time attacks)." Decision records let examiners follow the reasoning.

Maya's later decisions follow the same format. After the evaluation, one record adds a cumulative-change rule for the missed drift attack, with the alternative of a lower per-write threshold rejected because it flagged ordinary operator adjustments. Another records correlating HMI writes with operator login events, to address the setpoint write that came from allow-listed HMI-1.

38.3 Evaluation

The lab evaluates a monitor's output on eight labelled cases from Maya's synthetic set.

"""Lab 38: capstone evaluation - precision/recall with denominators (synthetic results)."""
cases = [("replay", True, True), ("replay", True, True), ("setpoint", True, False), ("setpoint", True, True),
         ("benign-maint", False, True), ("benign", False, False), ("benign", False, False), ("drift", True, False)]
tp = sum(a and d for _, a, d in cases); fp = sum((not a) and d for _, a, d in cases)
fn = sum(a and not d for _, a, d in cases); tn = sum((not a) and not d for _, a, d in cases)
print(f"TP={tp} FP={fp} FN={fn} TN={tn}")
print(f"precision={tp}/{tp+fp}={tp/(tp+fp):.2f} recall={tp}/{tp+fn}={tp/(tp+fn):.2f}")
print("missed classes:", sorted({c for c, a, d in cases if a and not d}))

Run with python3 lab_eval.py. Verified output:

TP=3 FP=1 FN=2 TN=2
precision=3/4=0.75 recall=3/5=0.60
missed classes: ['drift', 'setpoint']

The eight cases include five attacks and three benign writes, among them a maintenance write. The lab computes precision and recall from the monitor's labels and prints the confusion matrix. The set is a toy: its purpose is to show which classes the monitor misses, and a ninth case could change every rate. Its value is that each case has a known answer written before the monitor was run, so a disagreement is always the monitor's error or the label's, and Maya records which.

38.4 Reading the confusion matrix

Three true positives, one false positive (a maintenance write flagged), two false negatives (one setpoint attack and the drift attack missed), two true negatives. Precision 3/4 = 0.75; recall 3/5 = 0.60. Eight cases are far too few for precise rates — a Wilson interval for 3/5 spans roughly 0.23 to 0.88 — so the honest use of this table is to find failure classes, not to rank tools.

The single false positive is the maintenance write, which Decision 7 treats as benign only when its ticket ID appears in the window list. The two misses point to different fixes, a cumulative-change rule for the drift attack and login correlation for the HMI-1 write, which is why Maya examines each class separately rather than tuning one threshold.

Worked example (synthetic LDHP)

Maya examines the two misses. The drift attack used many small writes, each within the allow-listed range — a rule-based monitor without a rate or trend rule cannot see it. The missed setpoint attack came from HMI-1 itself, which is allow-listed. Both lead to new decision records: add a cumulative-change rule, and correlate HMI writes with operator login events.

Synthetic case — Lakeside District Heating Plant. The eight cases are a toy set for illustration.

Errors and limits

Labels are only as good as the scenario author; an attack labelled "benign" corrupts every metric. Do not tune rules on the evaluation cases and then report the same numbers.

Exercises

Synthesis

Implement in tested increments, record decisions, and use small evaluation sets to find failure classes rather than to claim rates.

Continue studying — chapter 38 of 40

← Chapter 37 (draft) · Course chapter index · Chapter 39 (draft) →