Chapter 5 — AI and machine-learning security: tools and interpretation
Lesson map
Lesson 05 published. Map, example, EX021.
flowchart TD
N0["Learning outcomes"]
N1["5.1 Harness and logs"]
N2["5.2 Rates and intervals"]
N3["5.3 Base rates"]
N4["5.4 Vendor tools"]
N5["5.5 Common misreadings"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX021"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["5.1 Harness and logs"]
N2["5.2 Rates and intervals"]
N3["5.3 Base rates"]
N4["5.4 Vendor tools"]
N5["5.5 Common misreadings"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX021"]
[Learning outcomes]
|
v
[5.1 Harness and logs]
|
v
[5.2 Rates and intervals]
|
v
[5.3 Base rates]
|
v
[5.4 Vendor tools]
|
v
[5.5 Common misreadings]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX021]
Learning outcomes
You will (1) use a simple test harness and logs; (2) interpret results with rates and intervals; (3) account for base rates in detection; (4) evaluate vendor security tools; and (5) avoid common misreadings.
5.1 Harness and logs
A harness runs test cases against a staging service, collects replies and tool logs, grades outcomes with rules and writes a report. Logs with request IDs let you see what the model read, what it proposed and what was executed.
At Meridian, Sofia's harness runs her ClaimChat suites against staging and grades each case by what the tool log shows. Keep the harness under version control with its suites, so results can be repeated after changes. Log every input, output and tool call with the case ID.
5.2 Rates and intervals
Report successes over trials with an interval. 0 in 100 still allows about 3%. A drop from 28% to 10% on 40 cases each may be real, but intervals overlap; more cases help.
Sofia's 11 of 40 urgent bookings had an interval of about 16–43%. After controls, her 0 of 90 left an upper bound of about 3%. Both are honest statements of what was tested. Choose the number of cases from the decision. To show a rate is below a few per cent, you need about a hundred trials with no failures; to show a large change, a few dozen per arm may be enough. Lab S lets you check intervals for your own counts, and the result should be written as counts, not only percentages.
5.3 Base rates
A fraud model with 95% detection and 5% false positives sounds strong. If 1 in 100 claims is fraudulent, only about 16% of flagged claims are fraud. Honest claimants bear the cost of the rest.
"""Lab B: base rates and alarm precision (Bayes' rule).
A detector for malicious prompts with given true- and false-positive rates
is applied to traffic with different shares of malicious prompts."""
for tpr, fpr in ((0.95, 0.05), (0.95, 0.01), (0.80, 0.001)):
for prev in (0.10, 0.01, 0.001):
tp = tpr * prev; fp = fpr * (1 - prev)
prec = tp / (tp + fp)
per_10k = round(10000 * (tp + fp))
print(f"tpr={tpr:.2f} fpr={fpr:.3f} prevalence={prev:.3f} precision={prec:.3f} alerts_per_10k={per_10k}")
Expected output:
tpr=0.95 fpr=0.050 prevalence=0.100 precision=0.679 alerts_per_10k=1400
tpr=0.95 fpr=0.050 prevalence=0.010 precision=0.161 alerts_per_10k=590
tpr=0.95 fpr=0.050 prevalence=0.001 precision=0.019 alerts_per_10k=509
tpr=0.95 fpr=0.010 prevalence=0.100 precision=0.913 alerts_per_10k=1040
tpr=0.95 fpr=0.010 prevalence=0.010 precision=0.490 alerts_per_10k=194
tpr=0.95 fpr=0.010 prevalence=0.001 precision=0.087 alerts_per_10k=109
tpr=0.80 fpr=0.001 prevalence=0.100 precision=0.989 alerts_per_10k=809
tpr=0.80 fpr=0.001 prevalence=0.010 precision=0.890 alerts_per_10k=90
tpr=0.80 fpr=0.001 prevalence=0.001 precision=0.445 alerts_per_10k=18
Lab B in this chapter shows how precision falls as fraud becomes rarer. For FraudFlag, Sofia used this to argue for fast human review of flagged claims, since most flagged claimants are honest and bear the delay. When the event is rare, even an accurate model produces many false alarms, so always ask how often the event occurs before judging a detector.
5.4 Vendor tools
Ask what threat the tool addresses; how it was evaluated (adaptively? by effects?); its false-positive rate on your traffic; what data it sends out; and whether it blocks or only monitors. Test it on your own suite before relying on it.
Sofia tested a vendor's injection filter on her 60 indirect cases and 100 normal uploads. It blocked 38 attacks but also 9 normal uploads. Run such a test before buying, and record both numbers. Ask the vendor for results on your own cases, not on theirs, and repeat the test after each update.
5.5 Common misreadings
"No attacks succeeded" from a small suite; "accuracy is high" ignoring base rates; "the filter blocks injection" based on vendor examples; "the model refused" while the tool still ran.
Each misreading overstates protection. Sofia's harness avoids the last one by checking tool logs, so a refusal with a booking still counts as a failure. Read each result for these four before reporting it. Ask a colleague to check your reading too, and keep the raw counts beside every summary so they can.
5.6 Worked example (synthetic Meridian case)
FraudFlag flags 6% of claims; fraud prevalence is about 1%. Sofia computes that most flagged claimants are honest and that each waits five extra days. She recommends reviewing flagged claims within 48 hours and telling claimants only that their claim is "under standard review". A vendor offers an injection filter; on her 60 indirect cases it blocks 38 but also blocks 9 of 100 normal uploads (handwritten quotes with unusual phrasing).
The vendor filter result gives both sides: 38 of 60 attacks blocked, 9 of 100 normal uploads blocked as well.
5.7 Errors and limits
- Small suites reported as proof.
- Ignoring false-positive costs.
- Trusting vendor demonstrations.
Each error leads to a wrong decision. A small suite with no failures still allows a meaningful failure rate. Ignoring false positives hides the cost to honest claimants waiting for FraudFlag reviews, or to customers whose normal uploads a filter blocks. Vendor demonstrations use favourable examples; Sofia's own suite gave a different picture. Check each result for all three before it reaches a decision. Fix any you find before reporting.
Exercises
- EX021 — Harness. List what a harness report should contain. Rubric: 3 = versions, suite sizes, rates with intervals, failures; 2 = three; 1 = one or two; 0 = none.
- EX022 — Base rate. Using the lab, find precision at 1% prevalence for 5% and 1% false positives. Rubric: 3 = 0.161 and 0.490 with meaning; 2 = numbers; 1 = one; 0 = wrong.
- EX023 — Vendor filter. Should Meridian buy the filter? Rubric: 3 = weighs 38/60 vs 9/100, monitoring role, not substitute for controls; 2 = partial; 1 = assertion; 0 = none.
- EX024 — Misreading. Correct the statement "the model refused, so the attack failed". Rubric: 3 = check tool logs; effect may occur despite refusal text; 2 = partial; 1 = vague; 0 = none.
- EX025 — Fairness. Why does the five-day delay matter for honest claimants? Rubric: 3 = most flagged are honest; harm falls on them; mitigation; 2 = partial; 1 = vague; 0 = none.
Synthesis
Tools produce numbers; practitioners interpret them with intervals, base rates and scepticism about vendor claims.
Continue studying — chapter 05 of 12
← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →