Chapter 12 — Probability and statistics

Lesson map

Lesson 12 published. Map, example, EX034.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["12.1 Conditional probability and base rates"]
  N2["12.2 Confusion matrices"]
  N3["12.3 Sampling and intervals"]
  N4["12.4 Simpson's paradox"]
  N5["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX034"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["12.1 Conditional probability and base rates"]
  N2["12.2 Confusion matrices"]
  N3["12.3 Sampling and intervals"]
  N4["12.4 Simpson's paradox"]
  N5["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX034"]

[Learning outcomes and connection]
    |
    v
[12.1 Conditional probability and base rates]
    |
    v
[12.2 Confusion matrices]
    |
    v
[12.3 Sampling and intervals]
    |
    v
[12.4 Simpson's paradox]
    |
    v
[Chapter synthesis]
    |
    v
[Worked example] --> [Exercise EX034]

Learning outcomes and connection

You will be able to (1) compute and interpret sensitivity, specificity and precision of a detection rule; (2) apply conditional probability and base rates; (3) explain how an aggregate rate can mislead (Simpson's paradox); and (4) report an interval rather than a bare proportion. Chapter 1 introduced uncertainty informally; here you quantify it. Chapter 15 applies these tools to response measures and Chapter 33 to outcome studies.

12.1 Conditional probability and base rates

P(A | B) is the probability of A given B. For detection, the key quantities are sensitivity P(alert | malicious), specificity P(no alert | benign) and precision (positive predictive value) P(malicious | alert). Bayes' theorem links them: precision depends on the base rate (prevalence) of malicious events. A rule with 95% sensitivity and 99% specificity, applied where only 1 in 1,000 events is malicious, produces about 10 false alerts for every true one.

The lab's base-rate line makes the point concrete: with sensitivity of 95 per cent, specificity of 99 per cent and one malicious event per thousand, fewer than one alert in ten is true. An analyst who knows this does not lose trust in a rule after a run of false alerts; the rule may be working exactly as its base rate predicts.

12.2 Confusion matrices

Count true positives (TP), false positives (FP), true negatives (TN) and false negatives (FN). Sensitivity = TP/(TP+FN); specificity = TN/(TN+FP); precision = TP/(TP+FP). Every one of these depends on how "malicious" was decided — usually by later investigation, which itself may miss cases (FNs are the hardest to count).

In Lab 12 each department's counts form its own matrix, and the precision of the old rule in Clinical, 8 true alerts out of 40, comes straight from it. False negatives in the lab are given; in practice they are found only when a missed event surfaces later, so they are almost always undercounted.

12.3 Sampling and intervals

A proportion from a sample has sampling error. The Wilson score interval is a well-behaved 95% interval for proportions and is easy to compute. Report "precision 0.30 (95% CI 0.21–0.41, 24 of 80 alerts)" rather than "30%".

The interval narrows as the sample grows. In Lab 12 the aggregate for the old rule rests on 100 alerts and its interval spans about twenty points; with five alerts, as in the exercise on intervals, the same method gives a span of more than sixty. Reporting the count beside the interval lets a reader see why. Use intervals for comparison with care: overlapping intervals do not by themselves show that two rules perform the same.

12.4 Simpson's paradox

An association in every subgroup can reverse when subgroups are combined, because subgroup sizes differ. Departments with high prevalence and many alerts can dominate an aggregate rate.

Lab 12 shows the effect with real arithmetic. The new rule is more precise in both Clinical and Admin, yet its overall precision falls, because in the new period most alerts came from Clinical, where malicious sign-ins are rarer. Nothing about the rule got worse; the mix of where alerts arose changed. Whenever an aggregate moves, ask whether the composition of the groups moved too, and report the subgroup figures beside the total so the reader can check.

12.5 Worked example and lab (synthetic): one rule, two departments

Save as lab12.py.

# Lab 12 - detection measures, base rates and Simpson's paradox (synthetic counts, stdlib only)
from math import sqrt

def wilson(k, n, z=1.96):
    if n == 0:
        return (float("nan"),) * 2
    p = k / n
    d = 1 + z*z/n
    c = p + z*z/(2*n)
    h = z * sqrt(p*(1-p)/n + z*z/(4*n*n))
    return ((c - h)/d, (c + h)/d)

# counts per department: (TP, FP, FN, TN) for rule R-SIGNIN over one quarter
DATA = {
    "Clinical": {"old": (8, 32, 2, 958), "new": (27, 63, 3, 907)},
    "Admin":    {"old": (45, 15, 5, 935), "new": (9, 1, 1, 89)},
}
def measures(tp, fp, fn, tn):
    return tp/(tp+fn), tn/(tn+fp), tp/(tp+fp)

for rule in ("old", "new"):
    tot = [0, 0, 0, 0]
    for dept, d in DATA.items():
        tp, fp, fn, tn = d[rule]
        tot = [a+b for a, b in zip(tot, d[rule])]
        sens, spec, prec = measures(tp, fp, fn, tn)
        print(f"{rule} {dept:8} prevalence {(tp+fn)/(tp+fp+fn+tn):.3f} precision {prec:.2f}")
    sens, spec, prec = measures(*tot)
    lo, hi = wilson(tot[0], tot[0]+tot[1])
    print(f"{rule} ALL      precision {prec:.2f} (95% CI {lo:.2f}-{hi:.2f}, {tot[0]} of {tot[0]+tot[1]} alerts)")

# base-rate illustration
sens, spec, prev = 0.95, 0.99, 0.001
ppv = sens*prev / (sens*prev + (1-spec)*(1-prev))
print(f"base-rate example: precision {ppv:.3f}")

Expected output:

old Clinical prevalence 0.010 precision 0.20
old Admin    prevalence 0.050 precision 0.75
old ALL      precision 0.53 (95% CI 0.43-0.62, 53 of 100 alerts)
new Clinical prevalence 0.030 precision 0.30
new Admin    prevalence 0.100 precision 0.90
new ALL      precision 0.36 (95% CI 0.27-0.46, 36 of 100 alerts)
base-rate example: precision 0.087

Interpretation: the new rule has higher precision in both departments (0.30 vs 0.20; 0.90 vs 0.75), yet the aggregate precision falls from 0.53 to 0.36. The reason is the mix: in the "new" period most alerts came from Clinical, where prevalence is low, while in the "old" period most came from Admin. A manager reading only the aggregate would wrongly conclude the new rule is worse. The correct comparison stratifies by department (or standardizes to a common mix). Note also that the two periods differ in more than the rule — departments' populations changed — so even the stratified comparison is not a controlled experiment. The base-rate line shows why rare-event detection yields mostly false alerts even with excellent specificity.

12.6 Common errors and limits

• Quoting sensitivity without the base rate, or precision without saying where it was measured.
• Aggregating heterogeneous groups.
• Treating "investigated as benign" as certainly benign; labels have error.

• Reporting proportions from tiny samples without intervals.

• Comparing two periods as if only the rule changed, when the populations and staffing changed too.

• Treating a narrow interval as proof that labels are correct; the interval covers sampling error, not labelling error.
• Computing precision on alerts that were investigated while ignoring those closed unread.

Exercises — chapter 12

Attempt each exercise before opening the worked results. Every case, organization, person, address and number in an exercise is synthetic unless a cited public source is named.

EX034 — Run and stratify. Steps: run Lab 12 and write a two-sentence conclusion a manager can use. Rubric: 3 = says the new rule is more precise within each department and the aggregate drop reflects mix change, with the caveat about uncontrolled periods; 2 = mentions only one; 1 = concludes the rule is worse; 0 = none.

EX035 — Bayes by hand. Inputs: sensitivity 0.90, specificity 0.98, prevalence 0.02. Steps: compute precision and explain what fraction of alerts an analyst should expect to be false. Rubric: 3 = precision ≈ 0.479, about 52% false; 2 = arithmetic slip; 1 = confuses sensitivity with precision; 0 = none.

EX036 — Interval reporting. Inputs: 3 true out of 5 alerts. Steps: compute the Wilson 95% interval with the lab function and rewrite "precision is 60%". Rubric: 3 = interval ≈ 0.23–0.88, sentence reports counts and width; 2 = interval right, sentence weak; 1 = no interval; 0 = none.

Chapter synthesis

Detection quality is conditional: precision depends on base rates, aggregates can reverse subgroup truths, labels carry error, and every proportion deserves its count and interval.