Chapter 12 — Probability and statistics
Lesson map
Lesson 12 published. Map, example, EX034.
flowchart TD
N0["Learning outcomes"]
N1["12.1 Rates"]
N2["12.2 Wilson interval"]
N3["12.3 Base rates"]
N4["12.4 Ranking defences"]
N5["12.5 Reporting"]
N6["12.7 Pulling a ranking slide"]
N7["12.8 Every interval in words"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX034"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["12.1 Rates"]
N2["12.2 Wilson interval"]
N3["12.3 Base rates"]
N4["12.4 Ranking defences"]
N5["12.5 Reporting"]
N6["12.7 Pulling a ranking slide"]
N7["12.8 Every interval in words"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX034"]
[Learning outcomes]
|
v
[12.1 Rates]
|
v
[12.2 Wilson interval]
|
v
[12.3 Base rates]
|
v
[12.4 Ranking defences]
|
v
[12.5 Reporting]
|
v
[12.7 Pulling a ranking slide]
|
v
[12.8 Every interval in words]
|
v
[Worked example] --> [Exercise EX034]
Learning outcomes
You will (1) compute a Wilson interval; (2) explain base rates; (3) avoid overclaiming from small n.
12.1 Rates
Detection rate k/n needs uncertainty. Seven of ten is not the same as seventy of one hundred.
Both give seventy per cent, but they support very different claims. With ten trials, a single extra miss or detection moves the rate by ten points, so the true rate could plausibly be anywhere from about forty to nearly ninety per cent. With a hundred trials the plausible range narrows to roughly sixty to seventy-eight. When the SOC reports that its new rule caught seven of the ten techniques Sam ran in a purple-team session, the honest summary is that the rule probably works more often than not, and that a stronger claim needs more trials.
12.2 Wilson interval
lab_wilson prints 95% intervals for synthetic purple-team detection counts.
"""Lab 12: Wilson 95% interval for a synthetic detection rate in a purple-team exercise."""
import math
def wilson(k, n, z=1.96):
p = k / n; d = 1 + z*z/n
c = (p + z*z/(2*n)) / d; h = z*math.sqrt(p*(1-p)/n + z*z/(4*n*n)) / d
return c - h, c + h
for k, n in [(7, 10), (70, 100)]:
lo, hi = wilson(k, n)
print(f"detected {k}/{n} rate={k/n:.2f} 95% CI=({lo:.2f}, {hi:.2f})")
Run with python3 lab_wilson.py. Verified output:
detected 7/10 rate=0.70 95% CI=(0.40, 0.89)
detected 70/100 rate=0.70 95% CI=(0.60, 0.78)
The Wilson interval is preferred to the textbook formula of p plus or minus 1.96 standard errors because that formula behaves badly with small samples and rates near zero or one. It can produce intervals below zero or above one hundred per cent, and it claims too much certainty when only a few trials were run. Wilson's version pulls the centre slightly towards one half and keeps the bounds inside the possible range. Purple-team exercises are exactly the small-sample, high-rate setting where that matters. Comparing the two printed lines shows the practical point: the same seventy per cent carries a range more than twice as wide at ten trials.
12.3 Base rates
Rare true issues drowned in noisy scanners waste fixer time.
Suppose a scanner flags a weakness correctly on ninety per cent of vulnerable hosts and wrongly on five per cent of patched ones, and only two per cent of TRB's thousand staging hosts are actually vulnerable. It will find about eighteen of the twenty real cases and raise about forty-nine false alarms, so roughly seven in ten flags are wrong. Developers who chase those false alarms learn to ignore the scanner. The base rate, the share of hosts that truly have the problem, decides how useful a positive result is, and that is why Sam confirms scanner output by hand before turning it into findings.
12.4 Ranking defences
Prefer intervals that do not overlap when claiming A beats B.
If one control stopped fourteen of twenty simulated phishing clicks and another stopped eleven of twenty, the first looks better, seventy against fifty-five per cent. Their Wilson intervals run from about forty-eight to eighty-five and from about thirty-four to seventy-four, which overlap widely, so the data cannot separate the two. Overlap does not prove the controls are equal. It shows that the experiment was too small to tell. Non-overlap is a rough, conservative rule: a formal test for the difference between two proportions is better, but the overlap check stops the most common overclaim before it reaches a slide.
12.5 Reporting
Managers hear counts and intervals in plain language.
A sentence such as detection was 0.7 with a 95% Wilson CI of 0.40 to 0.89 is correct and almost useless to a reader who has to decide something. Sam rewrites it as follows. The SOC detected seven of the ten techniques we tried. With so few trials, the real rate could be as low as four in ten or as high as nine in ten, so we recommend another round before changing the budget. The counts go first because people can picture them, the range comes next in words, and the sentence ends with what the uncertainty means for the decision.
12.6 Limits
Independence assumptions fail when tests share environment faults.
The interval calculations assume each trial is a separate draw, so that one result does not influence another. Real exercises break that assumption. If a log forwarder was down for an hour, every technique Sam ran in that hour went undetected for the same reason, and ten misses really amount to one fault counted ten times. The interval then looks narrower than the evidence justifies. Sam records the time and the target host for each trial, checks whether failures cluster, and reports a clustered failure as a single infrastructure finding rather than as a low detection rate.
12.7 Pulling a ranking slide
When intervals overlap, Sam forbids ranking defences as winners. A slide once claimed MFA beat phishing training on overlapping Wilson intervals; he pulled the slide and asked for more trials instead of a louder font.
The slide's author was not wrong that MFA probably helps more; prior knowledge suggests it does. The problem was that the slide presented the exercise data as proof, and the data could not carry that weight. Sam suggested wording that kept the prior and the evidence apart: industry experience favours MFA, and this exercise was too small to confirm or contradict that. He also proposed the trial count needed for a clearer answer next quarter.
12.8 Every interval in words
Sam requires every interval in a deck to appear once in words: "between about X and Y". Charts alone are not enough for board packs.
Error bars on a chart are easy to overlook and hard to read when slides are printed in greyscale or skimmed on a phone. A sentence survives both. The rule also catches mistakes, because writing that the rate was between about forty and ninety per cent makes a reviewer notice how wide that is, where a neat bar with a thin whisker does not. Board members who read the words consistently ask better questions, usually how to make the range narrower.
12.9 Planning the number of trials
A small tool suggests n for a target CI width. Sam uses it before promising detection rates to executives.
The planner inverts the interval: give it the rate you expect and how wide a range you can accept, and it returns roughly how many trials you need. For a rate near one half and a range of ten points either side, the answer is close to ninety-six trials. Running it before an exercise changes the conversation. A promise to measure detection to within ten points becomes either a commitment to about a hundred trials or an honest decision to report a wider range from the twenty trials the window allows.
12.10 Finance asks for one number
Finance asks for a single detection percentage for the annual report. Sam gives two sentences with intervals instead. Legal prefers his version because it cannot be read as a warranty. Uncertainty language protects the bank.
A single figure printed in an annual report becomes a public statement about the bank's defences. If TRB published that it detects seventy per cent of attacks, a regulator or litigant after a breach could treat that as a representation the bank failed to meet. Sam's sentences describe what was measured, in which exercise and with what uncertainty, which is accurate and much harder to misread. Finance gets a number to cite, and the bank does not promise more than its evidence shows.
12.11 Shrinking the claim
Sample size planner told Sam he needed n≈96 for a tight interval. He cut the claim instead of inventing data. Planners exist to shrink promises.
The exercise window allowed about thirty trials. Sam could have run the same techniques three times each to reach ninety-six, but repeated runs against the same hosts are not independent, so the narrower interval would have been false. Instead the report claimed what thirty independent trials supported: detection was probably above one half, and finer comparisons would need a larger exercise. The planner's main value was making that limit visible before anyone had promised more.
12.12 An appendix explaining the refusal
Board packs that insist on a single number get Sam's appendix with the refusal rationale. Legal endorsed the appendix.
The appendix is one page. It shows the counts, the interval and two short paragraphs. The first explains why a single percentage would overstate what the exercise measured. The second states what the bank can safely say. Writing it once saves arguing the same point every quarter, and attaching it whenever someone asks for a single figure moves the discussion from Sam's opinion to an agreed bank position. Legal's endorsement matters because the risk it addresses is legal as much as statistical.
12.13 Recording legal's endorsement
Refusal appendix has legal's endorsement email hash attached. Sam loves boring integrity.
The hash lets anyone later confirm that the appendix in a board pack is the version legal approved, not an edited copy. The endorsement email is stored in the governance archive, and its SHA-256 value is printed at the foot of the appendix. If the wording changes, the hash no longer matches and the change needs a new endorsement. This uses the same integrity habit as evidence handling: a document whose authority matters should be checkable without trusting whoever forwarded it.
Worked example (synthetic TRB)
Sam shows 7/10 CI (0.40,0.89) vs 70/100 (0.60,0.78) to explain why TRB funds more trials before claiming a detection win.
Synthetic case — Riverstone Transit Bank (TRB). The bank, Sam Ito, TrackScope/FindingsDesk/PulseMap and all figures are invented for teaching.
Errors and limits
This book grants no authority to test real banks or payment systems. Labs use documentation-range addresses and synthetic data only; exploit code is never provided.
Exercises
- EX034 — Interval read. What CI does the lab print for 7/10? Rubric: 3 = (0.40, 0.89); 2 = approximate; 1 = wrong; 0 = none.
- EX035 — Why wider. Why is the 7/10 interval wider than 70/100 at the same rate? Rubric: 3 = smaller sample / more uncertainty; 2 = partial; 1 = bug; 0 = none.
- EX036 — Manager sentence. Write one plain sentence Sam could say about 7/10. Rubric: 3 = count + uncertainty + not definitive; 2 = partial; 1 = "we are 70% safe"; 0 = none.
Synthesis
Small samples need intervals; confidence is not a security percentage.
Continue studying — chapter 12 of 40
← Chapter 11 (draft) · Course chapter index · Chapter 13 (draft) →