◇ OpenCyber Academy
Engineering · Chapter 41 · English

Chapter 41 — AI and machine-learning security: system optimization

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 41 — AI and machine-learning security: system optimization

Lesson map

Lesson 41 published. Map, example, EX121.

flowchart TD
  N0["Learning outcomes"]
  N1["41.1 Multi-objective optimisation"]
  N2["41.2 Threshold tuning"]
  N3["41.3 Placing controls"]
  N4["41.4 Goodhart's law"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX121"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["41.1 Multi-objective optimisation"]
  N2["41.2 Threshold tuning"]
  N3["41.3 Placing controls"]
  N4["41.4 Goodhart's law"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX121"]

[Learning outcomes]
    |
    v
[41.1 Multi-objective optimisation]
    |
    v
[41.2 Threshold tuning]
    |
    v
[41.3 Placing controls]
    |
    v
[41.4 Goodhart's law]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX121]

Learning outcomes

You will (1) optimise an AI system across security, safety, utility and cost; (2) use multi-objective thinking (Pareto fronts) rather than single metrics; (3) tune thresholds and control placement with data; and (4) avoid optimising security metrics into meaninglessness.

41.1 Multi-objective optimisation

Security improvements usually cost utility, latency or money. Plot alternatives on two or more axes (for example attack success versus task success) and identify the Pareto front: options where no other option is better on every axis. Decision makers then choose a point on the front using explicit priorities.

An option is dominated when another is at least as good on every axis and better on one; dominated options can be dropped without debate. The front itself usually contains several options, each giving up something different. Engineering can show the front clearly, but choosing between its points is a judgement about priorities, so it belongs with the accountable owners.

41.2 Threshold tuning

Detectors and anomaly models have thresholds. Choose them with a cost model: cost of a missed attack × its probability versus cost of a false alarm × its probability, at realistic base rates. Tune on validation data, never on the final test set, and revisit as base rates change.

Validation data must reflect the base rate expected in operation, or the chosen threshold will be wrong in service. A threshold set when attacks were rare may produce floods of alerts during a campaign, or miss too much if attackers adapt. Sam records the cost assumptions with each threshold, so a later change in base rate or in the cost of a miss prompts a deliberate retune.

41.3 Placing controls

Where a control sits changes its cost and effect. Running an injection classifier only on untrusted retrieved content (instead of every message) cuts cost while keeping coverage of the risky path. Placing confirmation only on high-impact actions reduces friction. Optimisation means putting strong controls on high-risk paths and light ones elsewhere.

The principle follows from the data-flow analysis: risk concentrates where untrusted input meets privileged action. Spending heavy controls everywhere wastes budget and annoys users without adding much protection, while light controls on the risky path leave it open. Moving the classifier to retrieved content in the performance chapter is an example: cost fell sharply while coverage of the untrusted path stayed the same.

41.4 Goodhart's law

When a measure becomes a target, it ceases to be a good measure. Teams that optimise a fixed jailbreak benchmark can overfit to it while real robustness does not improve. Keep held-out and refreshed suites, red-team periodically, and track several metrics.

A team rewarded for a falling attack-success number on one benchmark may add filters for its exact phrases, improving the score without making the system harder to attack. Held-out and refreshed suites break that link, since the team cannot tune to tests it has not seen. Tracking several metrics also helps, because gaming one usually worsens another, such as false refusals or task success.

41.5 Worked example (synthetic NRS case)

Sam evaluated five Wrench configurations on attack success (injection suite) and task success (planner suite): A (no controls) 31%/95%; B (classifier blocking) 9%/86%; C (policy + confirmation) 0.0% executions/93%; D (C + classifier flagging) 0.0%/93% with better reviewer focus; E (dual model + C) 0.0%/90%. B is dominated by C. D was chosen: same task success as C, plus flags that improved reviewer catch rates.

B blocks some attacks but costs the most task success, which explains why C dominates it: no executions and higher task success. E's dual model adds protection already provided by C at a further cost of three points. D adds flags on top of C at no task-success cost, which is why it wins.

41.6 Errors and limits

Optimising one number improves it at the expense of everything not measured. Thresholds tuned on test data look perfect in the report and disappoint in service. Benchmarks that never change are learned by the system and its developers, until a high score shows familiarity rather than robustness. Sam requires every optimisation report to show at least one security metric and one cost metric on a held-out set. Both appear on one chart.

Exercises

Synthesis

Optimisation in AI security is multi-objective: find the Pareto front, tune thresholds with cost models, place controls by risk, and guard against gaming your own metrics.

Continue studying — chapter 41 of 50

← Chapter 40 (draft) · Course chapter index · Chapter 42 (draft) →