◇ OpenCyber Academy
Bachelor's · Chapter 12 · English

Chapter 12 — Probability and statistics

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 12 — Probability and statistics

Lesson map

Lesson 12 published. Map, example, EX034.

flowchart TD
  N0["Learning outcomes"]
  N1["12.1 Zero is not zero"]
  N2["12.2 Wilson intervals"]
  N3["12.3 Reading the numbers"]
  N4["12.4 Base rates"]
  N5["12.5 Per-shift framing for operators"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX034"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["12.1 Zero is not zero"]
  N2["12.2 Wilson intervals"]
  N3["12.3 Reading the numbers"]
  N4["12.4 Base rates"]
  N5["12.5 Per-shift framing for operators"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX034"]

[Learning outcomes]
    |
    v
[12.1 Zero is not zero]
    |
    v
[12.2 Wilson intervals]
    |
    v
[12.3 Reading the numbers]
    |
    v
[12.4 Base rates]
    |
    v
[12.5 Per-shift framing for operators]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX034]

Learning outcomes

You will (1) estimate a false-alarm rate with a confidence bound; (2) use the rule of three for zero observed events; (3) compare Wilson intervals with naive point estimates; and (4) avoid the base-rate fallacy when evaluating alerts.

12.1 Zero is not zero

A detector that raised no false alarms in 40 quiet days has a point estimate of 0, but the honest statement is a bound. The rule of three gives an approximate 95 % upper limit of 3/n when zero events are seen in n independent trials; with n = 40 that is 7.5 %.

This is the claim the control engineer makes in this chapter's worked example: a new supply-temperature detector that "never false-alarms". Maya's first question was the denominator, 40 days, and her answer was the bound: 0 in 40 is consistent with a true daily false-alarm rate as high as several percent.

12.2 Wilson intervals

The Wilson score interval behaves well for small counts and proportions near 0 or 1, unlike the textbook normal approximation, which can produce negative lower bounds.

"""Lab 12: false-alarm estimates and rule-of-three bounds (synthetic counts)."""
import math
def wilson_upper(k, n, z=1.96):
    p = k / n; d = 1 + z*z/n
    return (p + z*z/(2*n) + z*math.sqrt(p*(1-p)/n + z*z/(4*n*n))) / d
for k, n in [(0, 40), (0, 300), (3, 300)]:
    r3 = 3 / n if k == 0 else float("nan")
    print(f"k={k:2} n={n:3} point={k/n:.4f} wilson95_upper={wilson_upper(k,n):.4f} rule_of_three={r3:.4f}")

Run with python3 lab_rule3.py. Verified output:

k= 0 n= 40 point=0.0000 wilson95_upper=0.0876 rule_of_three=0.0750
k= 0 n=300 point=0.0000 wilson95_upper=0.0126 rule_of_three=0.0100
k= 3 n=300 point=0.0100 wilson95_upper=0.0290 rule_of_three=nan

For LDHP's detector, the textbook interval for 0 false alarms in 40 days collapses to 0 % to 0 %, which states a certainty the data do not support. Wilson gives 0 % to 8.8 % for the same count, and that upper end is what Maya wrote in her note: "0 false alarms in 40 days; 95 % upper bound ≈ 8.8 % per day". The same method appears again in Chapter 25 for CUSUM's 0 of 40 false-alarm runs.

12.3 Reading the numbers

For 0/40 the Wilson upper bound (8.76 %) is a little above the rule of three (7.5 %); for 0/300 both shrink to about 1 %. Three false alarms in 300 gives a point estimate of 1 % but an upper bound near 2.9 %. Collecting more quiet days is the only way to narrow the interval.

Maya used the second row to answer the control engineer: about 300 quiet days would be needed to claim a false-alarm rate below roughly 1 %. At one day per day of operation, that is most of a year, which is why simulation studies such as Chapter 25's use many seeded runs instead.

12.4 Base rates

Suppose real attacks occur on 1 day in 1,000 and a detector catches 90 % of them while raising a false alarm on 2 % of quiet days. Of 1,000 days, it flags about 0.9 true and 19.98 false — so roughly 4 % of alerts are real. Operators who learn that most alerts are noise begin to ignore them, which is why alarm-management practice treats nuisance rates as a safety issue.

For LDHP's control room, where one night shift sees three or four genuine process alarms, a detector producing about 20 false alarms per real one would be switched off within a week.

Worked example (synthetic LDHP)

The control engineer says a new supply-temperature detector "never false-alarms". Maya asks for the denominator: 40 days. She writes: "0 false alarms in 40 days; 95 % upper bound ≈ 8.8 % per day (Wilson); about 300 quiet days would be needed to claim below roughly 1 %."

Synthetic case — Lakeside District Heating Plant. Counts are invented; the methods are standard.

12.5 Per-shift framing for operators

Operators think in shifts, not proportions. A 2 % per-day false-alarm rate on one detector sounds small, but forty such detectors produce almost one nuisance alarm per day. Convert rates into "expected alarms per shift" before asking a control room to accept a new detector.

At LDHP, with two twelve-hour shifts a day, a single detector at 2 % per day produces a nuisance alarm about once every 100 shifts, which operators will tolerate. Forty detectors at that rate produce about 0.8 per day, or roughly one every two or three shifts. The band detector in Chapter 25, which false-alarmed in 9 of 40 quiet runs, would be far worse on that scale.

Errors and limits

Bounds assume independent trials; consecutive days in the same weather regime are correlated, so the true uncertainty is wider.

Exercises

Synthesis

Always publish detection claims with a count, a denominator and a bound; and judge alerts by how often they are right, not just how often attacks are caught.

Continue studying — chapter 12 of 40

← Chapter 11 (draft) · Course chapter index · Chapter 13 (draft) →