Chapter 3 — Programming foundations | Technologist AI and machine-learning security

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 03 published. Map, example, EX007.

flowchart TD
  N0["Learning outcomes"]
  N1["3.1 Programs around models"]
  N2["3.2 Model output is untrusted"]
  N3["3.3 Validate structured output"]
  N4["3.5 Reading the lab"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX007"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["3.1 Programs around models"]
  N2["3.2 Model output is untrusted"]
  N3["3.3 Validate structured output"]
  N4["3.5 Reading the lab"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX007"]

[Learning outcomes]
    |
    v
[3.1 Programs around models]
    |
    v
[3.2 Model output is untrusted]
    |
    v
[3.3 Validate structured output]
    |
    v
[3.5 Reading the lab]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX007]

Learning outcomes

You will (1) write simple Python to call and test AI services safely; (2) validate untrusted input and model output; (3) handle errors and secrets correctly; and (4) explain why model output must be treated as untrusted data.

3.1 Programs around models

Most AI-security work is ordinary programming: building prompts, parsing responses, calling tools, writing tests. Python's standard library is enough for many tasks: json to parse structured output, re for patterns, hashlib for integrity checks, unittest for tests. Jonas's work on Ask Brightwell is mostly of this kind: escaping output in the staff console, validating the returns tool's JSON, and writing tests that fail when a cross-user lookup succeeds. None of it requires changing the model. Each is plain Python that a reviewer can read in minutes and check against the threat it addresses.

3.2 Model output is untrusted

A model's answer may contain text an attacker planted through a document. If code inserts that text into HTML, SQL or a shell command without escaping, the attacker gains injection into those systems — OWASP LLM05 Improper Output Handling. Treat output exactly like user input: escape for HTML, use parameterised queries, never pass it to a shell. Jonas's planted review in the staff console showed this directly: the assistant quoted an image tag from a review, and the console rendered it, running the script in a staff member's browser. The fix was escaping, not a better model.

3.3 Validate structured output

When a model must return an action, ask for JSON and validate it strictly: parse, check that only expected keys exist, check types, check allowed values (for example, action in {"lookup", "return"}), and reject anything else. Never "fix up" invalid output by guessing. For Ask Brightwell, the orchestrator accepts only lookup and return as actions and checks that the order number is a string of digits. Anything else, including extra keys the model invents, causes the request to be rejected and logged, so a malformed proposal never reaches the returns tool. The policy service then makes its own check on the validated action.

3.4 Errors and secrets

Fail closed: if validation fails or a tool errors, do not perform the action. Keep API keys in environment variables or a secrets manager, never in code or logs.

"""Lab E: evasion of a toy linear classifier (synthetic data, stdlib only).
Trains logistic regression on two synthetic clusters, then applies a
fast-gradient-sign (FGSM-style) perturbation of size eps to every test point
and reports clean vs adversarial accuracy. Teaching model only."""
import math, random
rng = random.Random(7)
def point(label):
    cx = 1.0 if label else -1.0
    return [rng.gauss(cx, 0.7), rng.gauss(cx, 0.7)], label
train = [point(i % 2) for i in range(400)]
test = [point(i % 2) for i in range(200)]
w, b = [0.0, 0.0], 0.0
sig = lambda z: 1 / (1 + math.exp(-z))
for epoch in range(200):
    gw, gb = [0.0, 0.0], 0.0
    for x, y in train:
        p = sig(w[0]*x[0] + w[1]*x[1] + b)
        gw[0] += (p - y)*x[0]; gw[1] += (p - y)*x[1]; gb += p - y
    w = [w[i] - 0.1*gw[i]/len(train) for i in range(2)]; b -= 0.1*gb/len(train)
def acc(data, eps):
    ok = 0
    for x, y in data:
        # gradient of the loss w.r.t. x is (p - y) * w; FGSM steps along its sign
        p = sig(w[0]*x[0] + w[1]*x[1] + b)
        xa = [x[i] + eps*math.copysign(1, (p - y)*w[i]) for i in range(2)]
        ok += (sig(w[0]*xa[0] + w[1]*xa[1] + b) >= 0.5) == bool(y)
    return ok / len(data)
print(f"weights w=({w[0]:.3f}, {w[1]:.3f}) b={b:.3f}")
for eps in (0.0, 0.25, 0.5, 0.75, 1.0):
    print(f"eps={eps:.2f} accuracy={acc(test, eps):.3f}")

Expected output:

weights w=(1.795, 1.756) b=-0.080
eps=0.00 accuracy=0.985
eps=0.25 accuracy=0.950
eps=0.50 accuracy=0.840
eps=0.75 accuracy=0.660
eps=1.00 accuracy=0.505

In Ask Brightwell, if the order tool times out, the assistant tells the customer it cannot look up the order now, rather than guessing. The vendor API key lives in the secrets manager, and the orchestrator's log filter removes it from any error message before writing. Jonas added a test that fails if a key-shaped string appears in a log line, so the rule is checked on every build.

3.5 Reading the lab

The lab trains a tiny linear classifier and shows its accuracy falling as inputs are pushed by eps along the worst direction: 0.985 at no perturbation, 0.505 at eps 1.0. It is a reminder that a model's decision can be steered by small, deliberate input changes; code around models must not assume their outputs are correct. For Brightwell, the lab supports a design choice rather than a fact about ReturnGuard: decisions that matter should not rest on a single score without checks. The perturbations are chosen by the attacker in the lab; real attackers face limits on what they can change, which is why the lab is a reminder, not a measurement.

3.6 Worked example (synthetic Brightwell case)

Ask Brightwell's staff console displayed model answers as raw HTML. Jonas plants a product review containing <img src=x onerror=...> on a test copy; the assistant quotes it and the script runs in the staff browser. He changes the console to escape output and adds a unit test. The fix sits where the data leaves the system, in the console, so it holds whatever the model says. The unit test plants the same image tag and fails if the console ever renders it as HTML again, which protects against a later change reintroducing the flaw.

3.7 Errors and limits

Inserting model text into HTML, SQL or shell.
Repairing invalid JSON by guesswork.
Secrets in source code.

Each of these turns a model's text into an action in another system. Ask Brightwell's console showed the first; strict validation of the returns tool's JSON avoids the second; and keeping the vendor key in the secrets manager avoids the third. Tests for each mean a later change cannot quietly bring the weakness back, which is the point of writing them as code.

Exercises

EX007 — Validator. Write pseudocode to validate {"action":..., "order_id":...}. Rubric: 3 = parse, exact keys, types, allowed values, fail closed; 2 = four; 1 = two; 0 = none.
EX008 — Output handling. Name three sinks where model output needs encoding and the right technique for each. Rubric: 3 = HTML escape, parameterised SQL, no shell/argument list; 2 = two; 1 = one; 0 = none.
EX009 — Lab. What does the drop to 0.505 at eps 1.0 mean? Rubric: 3 = near chance, small crafted changes flip decisions, toy caveat; 2 = partial; 1 = vague; 0 = wrong.

Synthesis

Programming around models means validating and encoding everything that crosses into other systems — especially model output — and failing closed.

Continue studying — chapter 03 of 30

← Chapter 02 (draft) · Course chapter index · Chapter 04 (draft) →