Chapter 32 — AI and machine-learning security: reliability
Lesson map
Lesson 32 published. Map, example, EX094.
flowchart TD
N0["Learning outcomes"]
N1["32.1 Reliability for AI"]
N2["32.2 Calibration"]
N3["32.3 Redundancy and degradation"]
N4["32.4 Reliability or attack?"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX094"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["32.1 Reliability for AI"]
N2["32.2 Calibration"]
N3["32.3 Redundancy and degradation"]
N4["32.4 Reliability or attack?"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX094"]
[Learning outcomes]
|
v
[32.1 Reliability for AI]
|
v
[32.2 Calibration]
|
v
[32.3 Redundancy and degradation]
|
v
[32.4 Reliability or attack?]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX094]
Learning outcomes
You will (1) define reliability for AI components (consistent correct behaviour over time and conditions); (2) quantify reliability with stratified error rates, calibration and availability; (3) design redundancy and graceful degradation; and (4) distinguish reliability failures from attacks in incident analysis.
32.1 Reliability for AI
Classical reliability asks how often a component fails. For AI, "failure" includes wrong outputs that look normal. Reliability therefore includes accuracy across conditions (stratified error rates), calibration (does 90% confidence mean 90% correct?), stability across versions and availability of the service.
A crashed service is noticed at once; a model that quietly misses more defects at dawn is not. Reliability measures for AI must therefore look at outputs over time and by condition, not just at uptime. Stability across versions matters because a model update can change decisions on cases that used to be right, and operators who learned to trust the old behaviour may not notice.
32.2 Calibration
A calibrated model's confidence matches its accuracy. Overconfident models make downstream automation brittle: if TrackEye says 99% "no defect" but is right only 95% of the time at that confidence, operators trust it too much. Measure calibration with reliability diagrams or expected calibration error; recalibrate with simple methods such as temperature scaling on held-out data.
Temperature scaling divides the model's raw scores by a single fitted number before converting them to probabilities. It does not change which class wins, so accuracy stays the same, but confidences move closer to observed accuracy. Calibration must be checked per condition stratum too, because a model can be well calibrated overall and badly overconfident at dawn.
32.3 Redundancy and degradation
Redundancy can be diverse (a different model or a rule-based check) or repeated (a second inference). Diverse redundancy helps against shared failure modes and some attacks, because an adversarial input crafted for one model may not transfer perfectly. Graceful degradation defines what happens when a component fails or is uncertain: TrackEye flags segments for manual review rather than marking them clear.
Repeated inference on the same model helps only against random faults, such as a transient hardware error, since the same input gives the same mistake.
32.4 Reliability or attack?
When error rates change, ask: did inputs change (drift), did the model change (version), or did someone act (attack)? Look at timing, affected strata, correlation with specific sources and logs. Many apparent attacks are drift; some apparent drift is an attack.
Evidence decides, not instinct. A drift typically starts gradually or at a known change, affects many inputs in a condition, and correlates with environment or version. An attack often starts abruptly, concentrates on particular sources or targets, and may avoid conditions that would make it obvious. Sam's triage checklist records each observation, so a later reviewer can see why an incident was classified one way and revisit it if new evidence appears.
32.5 Worked example (synthetic NRS case)
Pulse's false-negative rate rose on one route. Logs showed the rise coincided with a sensor-firmware update that changed units on two axles — a reliability failure, not an attack. Sam added unit checks at ingestion and a diverse rule-based threshold check that would have caught the extreme readings regardless of the model.
The timing pointed to the firmware update, and only the two affected axles changed, which fits a configuration change rather than an attacker. The fixes address both the cause and the class of failure: unit checks reject impossible values at once, and the rule-based check does not depend on the model at all.
32.6 Errors and limits
- Treating confidence scores as probabilities without calibration.
- Redundancy with identical models (shared failure).
- Jumping to "attack" without checking drift and changes.
Uncalibrated scores mislead anyone who sets thresholds or explains decisions with them. Identical redundant models fail together, so they add cost without adding safety. Calling every change an attack wastes investigation effort and can damage trust with suppliers and partners, while calling every change drift can let a real attack continue. The habit Sam teaches is to collect evidence for both explanations before deciding.
Exercises
- EX094 — Calibration. A model is 80% accurate on predictions where it reports 95% confidence. What is wrong and what is one fix? Rubric: 3 = overconfident; recalibrate (temperature scaling) and verify; 2 = diagnosis only; 1 = vague; 0 = wrong.
- EX095 — Diverse redundancy. Why is a rule-based check useful alongside Pulse? Rubric: 3 = different failure modes; adversarial/drift effects less likely to affect both; 2 = partial; 1 = vague; 0 = wrong.
- EX096 — Triage. List three questions to separate drift from attack. Rubric: 3 = what changed (inputs, versions), which strata/sources, timing and correlation with logs; 2 = two; 1 = one; 0 = none.
Synthesis
Reliable AI is calibrated, stratified, redundant in diverse ways and degrades safely; incident analysis separates drift from attack using evidence.
Continue studying — chapter 32 of 50
← Chapter 31 (draft) · Course chapter index · Chapter 33 (draft) →