Chapter 33 — AI and machine-learning security: safety and resilience | Engineering AI and machine-learning security

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 33 published. Map, example, EX097.

flowchart TD
  N0["Learning outcomes"]
  N1["33.1 Safe failure"]
  N2["33.2 Resilience"]
  N3["33.3 Effective human oversight"]
  N4["33.4 Misuse of generative capabilities"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX097"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["33.1 Safe failure"]
  N2["33.2 Resilience"]
  N3["33.3 Effective human oversight"]
  N4["33.4 Misuse of generative capabilities"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX097"]

[Learning outcomes]
    |
    v
[33.1 Safe failure]
    |
    v
[33.2 Resilience]
    |
    v
[33.3 Effective human oversight]
    |
    v
[33.4 Misuse of generative capabilities]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX097]

Learning outcomes

You will (1) design AI systems that remain safe when components fail or are attacked; (2) apply resilience concepts (anticipate, withstand, recover, adapt); (3) define human oversight that is effective rather than nominal; and (4) plan for misuse of generative capabilities.

33.1 Safe failure

Every AI decision path needs a defined safe state: manual inspection, a conservative default, refusing an action, or escalating to a human. Safety cases argue, with evidence, that hazards are controlled to acceptable levels — including hazards caused by adversaries.

For TrackEye, the safe state is manual inspection of the affected segment; for Wrench, it is refusing the tool action and telling the technician why. Choosing safe states is a design decision with costs: manual inspection uses staff time, and refusals slow work. The safety case states these costs and shows that they are acceptable, which also makes clear what would happen if an attacker deliberately forced the system into its safe state repeatedly.

33.2 Resilience

Resilience is the ability to anticipate, withstand, recover from and adapt to adverse conditions. For AI: anticipate with threat models and monitoring; withstand with layered controls; recover with rollback, retraining from verified data and credential revocation; adapt by updating tests and controls after incidents. The NIST AI RMF lists secure and resilient as a characteristic of trustworthy AI.

Recovery depends on preparation done earlier. Retraining from verified data is only possible if data versions and hashes were kept. Revoking credentials quickly needs a tested procedure and narrow credentials that can be revoked without stopping everything. Adaptation means each incident leaves a trace in the system: a new test in the suite, a changed threshold or an updated runbook, so the same failure is less likely next time.

33.3 Effective human oversight

Oversight fails when humans approve by habit ("automation bias") or lack information. Effective oversight shows the evidence behind a proposal (sources, confidence, what is unusual), limits volume so reviewers can concentrate, and tracks approval quality (for example, inserting known-bad test proposals to measure catch rates, with reviewers' consent and clear policy). The EU AI Act requires high-risk systems to be designed for effective human oversight (Article 14).

Known-bad proposals must be clearly identifiable afterwards so they never execute. Results are shared with reviewers as a group, never used to rank individuals.

33.4 Misuse of generative capabilities

Generative systems can be misused to produce harmful content or to assist attacks. NIST AI 600-1 discusses risks including information security and dangerous content, with suggested actions. Engineering responses include usage policies, abuse monitoring, refusal training, rate limits and restricting high-risk capabilities to authenticated, accountable users.

Restricting capabilities to authenticated users makes misuse attributable, which deters it and supports investigation. For Wrench, the risk is modest, since its tools are narrow, but a general text generator connected to internal documents could be asked to draft convincing phishing using real supplier names. Abuse monitoring looks for such patterns, and usage policies tell staff what is acceptable before anyone needs to enforce them.

33.5 Worked example (synthetic NRS case)

Sam measured that technicians approved 97% of Wrench's work-order proposals in a median of 4 seconds. With the technicians' agreement, a monthly exercise inserted clearly marked synthetic test proposals with wrong assets; only 60% were rejected. The team redesigned the approval screen to highlight assets outside the technician's usual routes and any instruction-like source text; the catch rate in the next exercise rose to 88%.

The high approval rate and four-second median were the warning signs, since nobody reads a source document in four seconds. A catch rate of 60% meant two in five wrong proposals would have been approved. The redesign made the important information visible instead of asking people to try harder, and the next exercise measured the effect rather than assuming it.

33.6 Errors and limits

Human review as a checkbox.
No defined safe state.
Recovery plans that assume clean data is available without having kept it.

A checkbox review creates the appearance of control while approving almost everything. Without a defined safe state, a failing component may continue in an unsafe mode or stop in an unplanned one. Recovery plans fail on the day if the clean data, signed models or credentials they rely on were never kept. Each of these is checked by exercise, not by reading the plan, because plans often assume things nobody has tested.

Exercises

EX097 — Safe states. Define safe states for TrackEye, Wrench and Pulse. Rubric: 3 = manual inspection, no action/escalate, conservative alert/manual check; 2 = two; 1 = one; 0 = none.
EX098 — Resilience. Give one action for each of anticipate, withstand, recover and adapt for Wrench. Rubric: 3 = four appropriate; 2 = three; 1 = one or two; 0 = none.
EX099 — Oversight. Why did inserting test proposals reveal a problem, and what made oversight better? Rubric: 3 = measured real catch rate (60%) vs assumed; highlighting unusual and source information; 2 = partial; 1 = vague; 0 = none.

Synthesis

Safety and resilience assume controls will sometimes fail. Safe states, recovery plans and measured, well-designed human oversight keep failures bounded.

Continue studying — chapter 33 of 50

← Chapter 32 (draft) · Course chapter index · Chapter 34 (draft) →