Chapter 4 — Programming I | Engineering AI and machine-learning security

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 4 — Programming I

Lesson map

Lesson 04 published. Map, example, EX010.

flowchart TD
  N0["Learning outcomes"]
  N1["4.1 Programs as precise instructions"]
  N2["4.2 Data structures"]
  N3["4.3 Untrusted input"]
  N4["4.4 Errors as information"]
  N5["4.5 Testing"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX010"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["4.1 Programs as precise instructions"]
  N2["4.2 Data structures"]
  N3["4.3 Untrusted input"]
  N4["4.4 Errors as information"]
  N5["4.5 Testing"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX010"]

[Learning outcomes]
    |
    v
[4.1 Programs as precise instructions]
    |
    v
[4.2 Data structures]
    |
    v
[4.3 Untrusted input]
    |
    v
[4.4 Errors as information]
    |
    v
[4.5 Testing]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX010]

Learning outcomes

You will (1) write small, correct Python programs with functions, loops and data structures; (2) handle untrusted input safely; (3) test code with assertions; and (4) read the book's labs and modify them.

4.1 Programs as precise instructions

A program states exactly what the computer does. Variables hold values; functions package steps; loops repeat; conditionals choose. Readable code uses clear names and small functions. The labs in this book use only Python's standard library so they run anywhere.

Precision cuts both ways: the computer does what the code says, not what the author meant. A loop that stops one item early or a condition written with 'or' instead of 'and' produces wrong results without any warning. Sam reads each lab line by line before running it and predicts the output, so a surprise in the printed result points to a gap in his understanding.

4.2 Data structures

Lists hold sequences; dictionaries map keys to values; sets hold unique items. A model inventory is naturally a list of dictionaries; a policy allowlist is naturally a set. Choosing the right structure makes code shorter and safer: membership in a set is a clear, fast test for "is this tool allowed?".

A list would also work for an allowlist, but checking membership in it means scanning every item, and duplicates can creep in unnoticed. The wrong structure can hide a security bug as well: if allowed tools are stored as one comma-separated string, a check such as 'search' in allowed would also accept a tool named 'research'.

4.3 Untrusted input

Any data from outside your program — files, network messages, model outputs — is untrusted. Never pass it to eval, shell commands or database queries without strict validation. This applies to AI: OWASP LLM 2025 lists improper output handling (LLM05) because applications often execute or render model output directly. Parse with strict parsers (e.g., json.loads), validate types and ranges, and reject unknown fields.

Rejecting unknown fields matters because downstream systems may act on fields the validator never looked at. A strict parser refuses them outright.

4.4 Errors as information

When code fails, it should fail closed and explain itself. A parser that silently drops an unknown field might let through a field the downstream system honours; a parser that rejects the whole object with a clear message is safer and easier to debug. Log the rejection with enough context to investigate (time, source, reason) but without copying sensitive content into logs. Exceptions are the normal Python mechanism: raise ValueError with a message and let the caller decide whether to retry, ask a human or stop.

4.5 Testing

An assert states what must be true. Unit tests call functions with chosen inputs and check outputs, including edge cases (empty lists, very large values, malformed text). Tests make later changes safe.

For security code the most valuable tests are the ones that should fail: a work order with an extra field, a priority spelled differently, an asset identifier containing punctuation. Each expected rejection is written as a test, so a later change that loosens validation breaks the build instead of reaching production. Sam keeps every malformed input that ever caused a problem as a permanent test case.

4.6 Worked example (synthetic NRS case)

Sam writes a function that parses Wrench's proposed work order (JSON) before it reaches the work-order system:

import json
ALLOWED = {"asset_id", "priority", "summary"}
def parse_order(text):
    d = json.loads(text)
    assert isinstance(d, dict) and set(d) <= ALLOWED, "unexpected fields"
    assert d.get("priority") in {"low", "medium", "high"}, "bad priority"
    assert isinstance(d.get("asset_id"), str) and d["asset_id"].isalnum(), "bad asset id"
    return d
print(parse_order('{"asset_id": "AX12", "priority": "high", "summary": "bearing hot"}'))
try:
    parse_order('{"asset_id": "AX12; rm", "priority": "high"}')
except AssertionError as e:
    print("rejected:", e)

The program prints the valid order, then rejected: bad asset id. The model's output is treated as data to validate, never as code.

The asset identifier is rejected because the semicolon and space fail the alphanumeric check, so text that looks like a shell command never reaches the work-order system. The function never requires a summary, so an order without one would pass; Sam adds that check next. In production Sam replaces the asserts with explicit exceptions, for the reason given among the errors below.

4.7 Errors and limits

Executing model output as code or shell commands.
Validation that only checks happy paths.
Using asserts as the only check in production code (they can be disabled with python -O; use explicit exceptions there).

Each error has a recognisable sign in code review. Calls to eval, exec or subprocess with a string built from model output are the first thing to search for. Validators with no tests for rejected inputs have probably never been tried on bad data. Asserts guarding security conditions should become if-statements that raise a named exception, because an optimised production run would otherwise skip the check entirely without any message.

Exercises

EX010 — Allowlist. Write a function allowed(tool) using a set of permitted tools. Rubric: 3 = correct set membership with test; 2 = works without test; 1 = uses list with errors; 0 = none.
EX011 — Output handling. Why is passing model output to eval dangerous? Name the OWASP entry. Rubric: 3 = arbitrary code execution from attacker-influenced text; LLM05; 2 = one part; 1 = vague; 0 = wrong.
EX012 — Edge cases. List four edge-case tests for parse_order. Rubric: 3 = four relevant (empty, extra field, wrong type, invalid JSON, long summary); 2 = three; 1 = one or two; 0 = none.

Synthesis

Programming for AI security starts with one habit: treat everything from outside — including model output — as untrusted data, validate it strictly, and test the edges.

Continue studying — chapter 04 of 50

← Chapter 03 (draft) · Course chapter index · Chapter 05 (draft) →