## Chapter 17 — Precision falls when the rare event gets rarer ### Learning outcomes After this chapter, you should be able to: 1. Compute the meaning of one printed precision from tp and fp. 2. Use the 0.019 / 509 line as the sharp example. 3. Keep Omar's alert choice in prose, beside the nine lines rather than inside them. ### 17.1 Three pairs of rates, and three prevalences inside each pair. The outer loop walks `(tpr, fpr)` through `(0.95, 0.05)`, `(0.95, 0.01)` and `(0.80, 0.001)`. Prevalence then steps across one tenth, one hundredth and one thousandth, and the cartesian product is nine prints. `tp = tpr * prev` and `fp = fpr * (1 - prev)`. Precision is `tp / (tp + fp)`. Alerts per 10,000 are `round(10000 * (tp + fp))`. Hold detection at 0.95 and the false-positive rate at 0.050. Prevalence 0.100 prints precision 0.679 and 1400 alerts. Prevalence 0.010 prints precision 0.161 and 590 alerts. Prevalence 0.001 prints precision 0.019 and 509 alerts. The detector did not get worse at recognising the rare class. The rare class got rarer, so most alarms are the false ones. The 0.019 line is the one to remember: about 98 percent of those alerts are not the event, which is the complement of that precision. The same slide appears in the other two groups. At fpr 0.010 the three precisions are 0.913, 0.490 and 0.087. At tpr 0.80 and fpr 0.001 they are 0.989, 0.890 and 0.445, with alerts 809, 90 and 18. No tenth combination is printed. Omar's choice, already written in the adoption prose, is to watch actions such as an external send or an unusual volume, rather than to alarm on every suspicious prompt. That sentence is not one of the nine lines. The lines are a reason for the sentence. | Group | Sharp low-prevalence line | |---|---| | fpr 0.050 | precision 0.019, 509 alerts | | fpr 0.010 | precision 0.087, 109 alerts | | fpr 0.001 | precision 0.445, 18 alerts | ### 17.2 Worked example | Note | Decision | |---|---| | "At 0.1 percent prevalence and fpr 0.05, precision is 0.019." | Keep | | "509 alerts per 10,000 sit on that same line." | Keep | | "Add a row for fpr 0.02." | Refuse | | "Alarm on every odd prompt because the detector is 95 percent." | Refuse; the 0.019 line is why that alarm is mostly noise | > **Synthetic case.** Prevalence here is an input to the formula. It is not a count of Tallis incidents. ### Errors and limits A high true-positive rate does not protect precision when the event is rare and the false-positive rate is not tiny. ### Apply - **EX063 — Nine lines.** Walk the printed precisions and say what the 0.019 / 509 line changes about an alert on every suspicious prompt. *Rubric:* 3 = the nine lines as printed; 0.019 means almost all of those 509 alerts are false; watch actions instead; do not add a row;; 2 = the sharp line and the action decision;; 1 = "precision falls";; 0 = adds a tenth row. ### Continue studying — lesson 17 (chapter 17 of 20) [← Chapter 16](#ch-16) · [Course chapter index](../SUBJECTS.md) · [Chapter 18 →](#ch-18) Continue to the next authored lesson. ## Worked answers **EX063.** Nine lines are printed. With tpr 0.95 and fpr 0.050, precision moves from 0.679 to 0.161 to 0.019 as prevalence falls, and the last of those lines has 509 alerts. Precision 0.019 means almost all of those alerts are false, so the plan watches actions such as external sends rather than every suspicious prompt. A tenth row is not part of the print.