Chapter 36 — AI and machine-learning security: performance and scale
Lesson map
Lesson 36 published. Map, example, EX106.
flowchart TD
N0["Learning outcomes"]
N1["36.1 Security controls cost performance"]
N2["36.2 Unbounded consumption"]
N3["36.3 Sizing quotas"]
N4["36.4 Scale and control effectiveness"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX106"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["36.1 Security controls cost performance"]
N2["36.2 Unbounded consumption"]
N3["36.3 Sizing quotas"]
N4["36.4 Scale and control effectiveness"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX106"]
[Learning outcomes]
|
v
[36.1 Security controls cost performance]
|
v
[36.2 Unbounded consumption]
|
v
[36.3 Sizing quotas]
|
v
[36.4 Scale and control effectiveness]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX106]
Learning outcomes
You will (1) relate performance engineering (latency, throughput, cost) to security controls; (2) defend against unbounded consumption and denial of service; (3) size quotas and budgets; and (4) keep controls effective at scale.
36.1 Security controls cost performance
Every guardrail, classifier, second model and approval step adds latency and cost. Under load, teams are tempted to disable controls. Design controls with performance budgets from the start and measure them like any other feature.
A budget makes the trade-off explicit: for example, guardrails may add at most 150 ms to a Wrench reply. If a control exceeds its budget, the team optimises it or changes where it runs, rather than removing it under pressure. Measuring controls continuously also shows when one slowly becomes more expensive, as a classifier might when documents grow longer.
36.2 Unbounded consumption
OWASP LLM10 covers attacks that exhaust resources or budgets: extremely long inputs, prompts that cause very long outputs, recursive agent loops, and high request volumes. Controls: input size limits, output token caps, per-request compute timeouts, maximum agent steps, per-client token quotas, and cost alerts. The token-bucket lab shows how rate and burst settings trade service for normal users against protection from floods.
"""Lab R: token-bucket limits against unbounded consumption (synthetic).
Simulates 60 seconds of requests from a normal client and a flooding client
against a model endpoint with a per-client token bucket."""
import random
rng = random.Random(4)
def simulate(rate_per_s, burst, reqs):
tokens, last, ok = burst, 0.0, 0
for t in reqs:
tokens = min(burst, tokens + (t - last) * rate_per_s); last = t
if tokens >= 1: tokens -= 1; ok += 1
return ok
normal = sorted(rng.uniform(0, 60) for _ in range(90))
flood = sorted(rng.uniform(0, 60) for _ in range(3000))
for rate, burst in ((5, 20), (2, 10), (1, 5)):
print(f"limit={rate}/s burst={burst:2d} normal_served={simulate(rate, burst, normal):3d}/90 "
f"flood_served={simulate(rate, burst, flood):4d}/3000")
Expected output:
limit=5/s burst=20 normal_served= 90/90 flood_served= 319/3000
limit=2/s burst=10 normal_served= 90/90 flood_served= 129/3000
limit=1/s burst= 5 normal_served= 62/90 flood_served= 64/3000
Each control caps a different dimension, and attackers will find whichever one is missing. Wrench's runaway loop showed the step limit missing.
36.3 Sizing quotas
Use observed usage percentiles (for example the 99th percentile of tokens per user per hour) plus headroom, and differentiate tiers. Re-tune as usage changes. Combine quotas with anomaly detection, since attackers can stay just below fixed limits.
The lab's third setting shows the risk of guessing: a limit of 1 per second with a burst of 5 refuses 28 of 90 normal requests. Percentile-based sizing avoids that by starting from what real users do. Tiers let a heavy but legitimate integration have a larger quota without raising everyone's. An attacker who learns the limit can stay just under it, so anomaly detection on patterns, such as the distinct-input ratio, covers what fixed limits cannot.
36.4 Scale and control effectiveness
At scale, rare events become common: a 0.1% injection success rate over 10 million documents per month is 10,000 successes. Controls that rely on human review must be sized for that volume, or be designed so that each success has bounded impact.
Ten thousand successes a month cannot each be reviewed carefully by a small team, so the design must make each one harmless on its own: a tainted proposal that needs confirmation, a tool that can comment but not close. Rates acceptable in a pilot can become unacceptable in production purely because of volume, so Sam recomputes expected counts whenever usage grows.
36.5 Worked example (synthetic NRS case)
A misbehaving integration caused Wrench to loop: each tool result triggered another search, reaching 400 steps on one task and consuming a day's budget. Sam set a maximum of 12 steps per task, a per-task token cap and a cost alert at 150% of the daily median. The injection classifier was moved to run only on retrieved content rather than on every message, cutting its cost by 60% without reducing coverage of the untrusted path.
Moving the classifier is safe because the untrusted path is where injections arrive; messages typed by authenticated technicians are not the risk it was built for.
36.6 Errors and limits
- Request-count limits that ignore token and compute cost.
- Disabling controls under load.
- Human-review designs that cannot scale with volume.
Counting requests alone lets one huge prompt cost as much as thousands of small ones. Disabling controls under load creates exactly the conditions an attacker would choose, since floods can be used to provoke it. Review designs that assume a fixed number of cases break when volume grows. Sam's operations review asks, for each control, what its cost is per request and what happens to it at ten times today's traffic.
Exercises
- EX106 — Lab reading. Which setting in the lab protects against the flood while fully serving the normal client with the tightest limit? Rubric: 3 = 2/s burst 10: 90/90 normal, 129/3000 flood; 2 = setting only; 1 = vague; 0 = wrong.
- EX107 — Scale arithmetic. At 0.05% success over 4 million documents per month, how many successes per month? Rubric: 3 = 2,000 with implication; 2 = number only; 1 = slip; 0 = none.
- EX108 — Loop limits. List four limits for agent loops. Rubric: 3 = max steps, per-task tokens, wall-time, cost alerts/kill switch; 2 = three; 1 = one or two; 0 = none.
Synthesis
Performance and security are coupled. Budgeted controls, compute-aware quotas and loop limits defend against unbounded consumption and keep protection affordable at scale.
Continue studying — chapter 36 of 50
← Chapter 35 (draft) · Course chapter index · Chapter 37 (draft) →