OpenCyber AcademyLesson map
Lesson 07 published. Map, example, EX019.
flowchart TD
N0["Learning outcomes"]
N1["7.1 The base-rate effect"]
N2["Lab 3 — base rate"]
N3["7.2 Reading the lab"]
N4["7.3 Calibrated words"]
N5["7.4 Skewed data"]
N6["Practice note — Recall and coverage"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX019"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["7.1 The base-rate effect"]
N2["Lab 3 — base rate"]
N3["7.2 Reading the lab"]
N4["7.3 Calibrated words"]
N5["7.4 Skewed data"]
N6["Practice note — Recall and coverage"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX019"]
[Learning outcomes]
|
v
[7.1 The base-rate effect]
|
v
[Lab 3 — base rate]
|
v
[7.2 Reading the lab]
|
v
[7.3 Calibrated words]
|
v
[7.4 Skewed data]
|
v
[Practice note — Recall and coverage]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX019]
Learning outcomes
You will (1) compute precision from sensitivity, false-positive rate and prevalence; (2) explain the base-rate effect on alerting; (3) express judgments with calibrated probability words; (4) choose summaries suited to skewed data.
7.1 The base-rate effect
A rule that matches 95% of malicious connections and wrongly matches 2% of benign ones sounds excellent. But if only 1 in 100 connections is malicious, most alerts are false. If only 1 in 1,000 is malicious, almost all are.
The lab makes the point with numbers. At the lower prevalence almost two thousand benign connections are flagged for under a hundred malicious ones. Analysts quickly learn to ignore such alerts, and then they miss the true ones. The rule itself has not changed between the two rows; only the traffic it runs against has. That is why Kofi always asks where a rule will run before judging how good it is.
Lab 3 — base rate
"""Lab 3 (ch 07): base-rate effect on alert precision for a synthetic indicator match rule."""
for prevalence in (0.01, 0.001):
sens, fpr, n = 0.95, 0.02, 100000
bad = n * prevalence; good = n - bad
tp, fp = bad * sens, good * fpr
print(f"prevalence {prevalence:.3f}: TP={tp:.0f} FP={fp:.0f} precision={tp/(tp+fp):.3f}")
Run with python3 lab_bayes.py. Verified output:
prevalence 0.010: TP=950 FP=1980 precision=0.324
prevalence 0.001: TP=95 FP=1998 precision=0.045
7.2 Reading the lab
At prevalence 0.010, 950 true and 1,980 false positives give precision 0.324. At 0.001, precision falls to 0.045: about 21 false alerts for each true one. Engineering response: lower the false-positive rate, raise prevalence by applying the rule only where malicious traffic is likelier, or route matches to enrichment before analysts.
The third option is what the worked example recommends. A match then adds weight to other evidence instead of paging an analyst on its own. Kofi also checks the arithmetic by hand once, because a lab that disagrees with a simple calculation usually has a bug.
7.3 Calibrated words
Analysts translate probabilities into words. Kent's essay on words of estimative probability showed readers interpret "probable" inconsistently; teams should publish a mapping. ICD 203 also asks that uncertainty be expressed properly.
Lantern's reporting guide gives each word a range: for example, likely means a probability between about 0.55 and 0.80. Analysts choose a word from the table instead of from instinct, and a reader can look up what it means. Kofi checks drafts for words outside the table, such as possibly, which readers interpret anywhere from rare to even chance. The table is reviewed each year with the people who read the reports, so the words keep the meaning readers expect.
7.4 Skewed data
Time-to-block and enrichment latency are right-skewed. Medians and percentiles (p90, p99) describe them better than means.
Most indicators are blocked within minutes, but a few wait for hours because a partner held them or an approval was slow. Those few pull the mean far above what a typical indicator experiences. The median shows the usual case and the ninetieth percentile shows how bad the slow cases get. Kofi reports both, with the count of items, so readers can see the long tail without it distorting the typical figure. A single mean would describe no actual indicator.
Practice note — Recall and coverage
Precision answers how many alerts are true; recall answers how many malicious events were caught. A feed that is very precise but covers a small part of the threat landscape may still leave BEN exposed. Lantern therefore reports both, and estimates recall from incident reviews where the full set of malicious events was later established. Kofi also keeps a small table of the assumed prevalence for each traffic segment, because the same rule behaves very differently on the guest Wi-Fi than on the substation management network.
Worked example
Kofi evaluates a proposed rule matching any connection to a feed's IPs. He estimates prevalence near 0.001 in BEN's traffic, computes precision 0.045, and recommends using the rule as an enrichment input rather than a direct page to analysts.
Synthetic case. Rates are illustrative.
Errors and limits
Prevalence is hard to know; state assumptions. Precision without recall hides missed threats.
Exercises
- EX019 — Precision quote. What precision does the lab give at prevalence 0.001?
Show marking criteria
Rubric: 3 = 0.045; 2 = approx; 1 = 0.95; 0 = none. - EX020 — Engineering fix. Name two ways to improve precision for this rule.
Show marking criteria
Rubric: 3 = two of lower FPR / scope to higher-prevalence segments / enrichment gate; 2 = one; 1 = raise sensitivity only; 0 = none. - EX021 — Percentile choice. Why report p90 enrichment latency instead of mean?
Show marking criteria
Rubric: 3 = skewed tail matters for user experience/SLA; mean hides it; 2 = vague; 1 = mean better; 0 = none.
Synthesis
Base rates dominate alert quality, words need numbers behind them, and skewed metrics need percentiles.
Continue studying — chapter 07 of 50
← Chapter 06 (draft) · Course chapter index · Chapter 08 (draft) →