Chapter 12 — Probability and statistics
Lesson map
Lesson 12 published. Map, example, EX034.
flowchart TD
N0["Learning outcomes"]
N1["12.1 Words of estimative probability"]
N2["Lab — checking estimative words"]
N3["12.2 Reading the result"]
N4["12.3 Base rates"]
N5["12.4 False precision"]
N6["12.5 Likelihood versus confidence"]
N7["12.6 Small samples"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX034"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["12.1 Words of estimative probability"]
N2["Lab — checking estimative words"]
N3["12.2 Reading the result"]
N4["12.3 Base rates"]
N5["12.4 False precision"]
N6["12.5 Likelihood versus confidence"]
N7["12.6 Small samples"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX034"]
[Learning outcomes]
|
v
[12.1 Words of estimative probability]
|
v
[Lab — checking estimative words]
|
v
[12.2 Reading the result]
|
v
[12.3 Base rates]
|
v
[12.4 False precision]
|
v
[12.5 Likelihood versus confidence]
|
v
[12.6 Small samples]
|
v
[Worked example] --> [Exercise EX034]
Learning outcomes
You will (1) express analytic confidence with calibrated words; (2) check wording against a probability scale; (3) explain base rates; (4) avoid false precision.
12.1 Words of estimative probability
Sherman Kent showed that readers interpret "probable" anywhere from roughly 20% to 90%. ICD 203 requires analytic products to express likelihood consistently. Teams adopt a scale — for example "very likely" for 80–95% — and stick to it.
Amara uses the team's scale in every product and adds it as a footnote, so readers outside the team can interpret the words. Kent's finding shows that the problem is not careless writing but genuinely different readings of the same words. A shared scale gives the words a fixed meaning, and the footnote lets readers at HSSC or HCHT's board apply it the same way.
Lab — checking estimative words
"""Lab 3 (ch 12): map estimative words to a published probability scale and flag misuse (synthetic)."""
SCALE = {"almost no chance": (0, 5), "very unlikely": (5, 20), "unlikely": (20, 45),
"roughly even chance": (45, 55), "likely": (55, 80), "very likely": (80, 95),
"almost certainly": (95, 100)}
claims = [("C1", "likely", 70), ("C2", "very likely", 60), ("C3", "unlikely", 30), ("C4", "probable", 75)]
for cid, word, analyst_pct in claims:
if word not in SCALE:
print(cid, f"'{word}' not on team scale -> rewrite")
continue
lo, hi = SCALE[word]
print(cid, word, f"{analyst_pct}%", "consistent" if lo <= analyst_pct <= hi else f"INCONSISTENT (scale {lo}-{hi}%)")
Run with python3 lab_wep.py. Verified output:
C1 likely 70% consistent
C2 very likely 60% INCONSISTENT (scale 80-95%)
C3 unlikely 30% consistent
C4 'probable' not on team scale -> rewrite
12.2 Reading the result
Claim C1 says "likely" at 70% — consistent. C2 says "very likely" but assigns 60%, outside the team's 80–95% band, so the wording overstates confidence. C3 "unlikely" at 30% is consistent. C4 uses "probable", which is not on the team scale and must be rewritten.
Amara returns the draft with the two inconsistent claims marked and the scale attached. The author can either change the word or change the number, but must explain which judgment is correct. Consistency is checked before anything leaves the team, because readers outside it will act on the word they see, not on a number they never read.
12.3 Base rates
If one email in ten thousand is malicious, a filter flagging 1% of email produces mostly false alarms even if it catches every phish. Always ask about the base rate before judging a detection.
Amara applies this to an alert rule proposed after an HSSC report and finds that most of its expected alerts would be harmless. The team narrows the rule before deploying it. After narrowing, the rule fires a few times a week, and each alert is checked within the day, which the team can sustain.
12.4 False precision
"There is a 73.4% chance this group is responsible" implies measurement that does not exist. Use bands.
Amara replaces such figures in drafts with a word from the team's scale and a short reason. A precise number suggests that the probability was calculated from data, when in practice it reflects an analyst's judgment based on incomplete evidence. Bands are honest about that and easier for readers to act on. If a number is needed for a model, it is given as a range and labelled as an estimate.
12.5 Likelihood versus confidence
Likelihood describes how probable an event is; confidence describes the quality of the evidence and reasoning behind the judgment. ICD 203 asks analysts to express and explain both.
Amara learns to write both in one sentence, such as "likely, with moderate confidence because only one source reports it". The two can differ: an event can be likely while confidence is low, if the evidence is thin. Readers need both to decide how much weight to give a judgment. Amara checks every judgment in her drafts for both parts.
12.6 Small samples
With five reports, one more or one fewer mention swings a percentage by twenty points. When samples are small, report counts, avoid ranking close values and say that the pattern could change with more data. Statistics cannot rescue a sample that simply contains too little information.
Amara applies this to HSSC's quarterly summary. Several quarters contain only a handful of reports per category, so she asks HSSC to show counts beside each percentage and to mark rankings that depend on one or two reports. HSSC adopts the change for the next summary.
Worked example
Amara drafts: "It is probable the phishing wave will reach HCHT this week." Her script flags "probable" as off-scale. She rewrites: "It is likely (55–80%) HCHT staff will receive emails from this wave this week; moderate confidence, based on HSSC reports from three nearby trusts."
Synthetic case. The draft and reports are invented.
Errors and limits
Scales reduce ambiguity but do not make judgments correct. Track outcomes to check calibration over time.
Exercises
- EX034 — Lab inconsistency. Which claim was inconsistent in the lab and why? Rubric: 3 = C2: "very likely" at 60%, scale 80–95%; 2 = C2 only; 1 = C4; 0 = none.
- EX035 — Base-rate trap. Explain why a filter flagging 1% of email might produce mostly false alarms. Rubric: 3 = malicious base rate far below 1%; most flags benign; 2 = partial; 1 = filter broken; 0 = none.
- EX036 — Two dimensions. Distinguish likelihood from confidence. Rubric: 3 = probability of event vs quality of evidence/reasoning; 2 = partial; 1 = same; 0 = none.
Synthesis
Use a shared scale, check base rates, avoid false precision and state both likelihood and confidence.
Continue studying — chapter 12 of 40
← Chapter 11 (draft) · Course chapter index · Chapter 13 (draft) →