OpenCyber AcademyLesson map
Lesson 17. Map, example, EX063.
flowchart TD
T["tp = tpr times prevalence"] --> P["precision = tp / (tp + fp)"]
F["fp = fpr times (1 - prevalence)"] --> P
P --> A["alerts = round of 10000 times (tp + fp)"]
Nine lines are the whole experiment. A tenth line would be an invention.
Logical drawing
tpr=0.95 fpr=0.050 prevalence=0.100 precision=0.679 alerts_per_10k=1400
tpr=0.95 fpr=0.050 prevalence=0.010 precision=0.161 alerts_per_10k=590
tpr=0.95 fpr=0.050 prevalence=0.001 precision=0.019 alerts_per_10k=509
tpr=0.95 fpr=0.010 prevalence=0.100 precision=0.913 alerts_per_10k=1040
tpr=0.95 fpr=0.010 prevalence=0.010 precision=0.490 alerts_per_10k=194
tpr=0.95 fpr=0.010 prevalence=0.001 precision=0.087 alerts_per_10k=109
tpr=0.80 fpr=0.001 prevalence=0.100 precision=0.989 alerts_per_10k=809
tpr=0.80 fpr=0.001 prevalence=0.010 precision=0.890 alerts_per_10k=90
tpr=0.80 fpr=0.001 prevalence=0.001 precision=0.445 alerts_per_10k=18
Learning outcomes
After this chapter, you should be able to:
- Compute the meaning of one printed precision from tp and fp.
- Use the 0.019 / 509 line as the sharp example.
- Keep Omar's alert choice in prose, beside the nine lines rather than inside them.
17.1 Three pairs of rates, and three prevalences inside each pair.
The outer loop walks (tpr, fpr) through (0.95, 0.05), (0.95, 0.01) and (0.80, 0.001). Prevalence then steps across one tenth, one hundredth and one thousandth, and the cartesian product is nine prints. tp = tpr * prev and fp = fpr * (1 - prev). Precision is tp / (tp + fp). Alerts per 10,000 are round(10000 * (tp + fp)).
Hold detection at 0.95 and the false-positive rate at 0.050. Prevalence 0.100 prints precision 0.679 and 1400 alerts. Prevalence 0.010 prints precision 0.161 and 590 alerts. Prevalence 0.001 prints precision 0.019 and 509 alerts. The detector did not get worse at recognising the rare class. The rare class got rarer, so most alarms are the false ones. The 0.019 line is the one to remember: about 98 percent of those alerts are not the event, which is the complement of that precision.
The same slide appears in the other two groups. At fpr 0.010 the three precisions are 0.913, 0.490 and 0.087. At tpr 0.80 and fpr 0.001 they are 0.989, 0.890 and 0.445, with alerts 809, 90 and 18. No tenth combination is printed.
Omar's choice, already written in the adoption prose, is to watch actions such as an external send or an unusual volume, rather than to alarm on every suspicious prompt. That sentence is not one of the nine lines. The lines are a reason for the sentence.
Lesson table; scroll horizontally to read all columns →
| Group | Sharp low-prevalence line |
|---|---|
| fpr 0.050 | precision 0.019, 509 alerts |
| fpr 0.010 | precision 0.087, 109 alerts |
| fpr 0.001 | precision 0.445, 18 alerts |
17.2 Worked example
Lesson table; scroll horizontally to read all columns →
| Note | Decision |
|---|---|
| "At 0.1 percent prevalence and fpr 0.05, precision is 0.019." | Keep |
| "509 alerts per 10,000 sit on that same line." | Keep |
| "Add a row for fpr 0.02." | Refuse |
| "Alarm on every odd prompt because the detector is 95 percent." | Refuse; the 0.019 line is why that alarm is mostly noise |
Synthetic case. Prevalence here is an input to the formula. It is not a count of Tallis incidents.
Errors and limits
A high true-positive rate does not protect precision when the event is rare and the false-positive rate is not tiny.
Apply
- EX063 — Nine lines. Walk the printed precisions and say what the 0.019 / 509 line changes about an alert on every suspicious prompt.
Show marking criteria
Rubric: 3 = the nine lines as printed; 0.019 means almost all of those 509 alerts are false; watch actions instead; do not add a row;; 2 = the sharp line and the action decision;; 1 = "precision falls";; 0 = adds a tenth row.