Chapter 3 — Programming I | Bachelor's AI and machine-learning security

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 03 published. Map, example, EX007.

flowchart TD
  N0["Learning outcomes"]
  N1["3.1 Why programming matters for AI security"]
  N2["3.2 Core constructs"]
  N3["3.3 Handling input safely"]
  N4["3.4 Testing"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX007"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["3.1 Why programming matters for AI security"]
  N2["3.2 Core constructs"]
  N3["3.3 Handling input safely"]
  N4["3.4 Testing"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX007"]

[Learning outcomes]
    |
    v
[3.1 Why programming matters for AI security]
    |
    v
[3.2 Core constructs]
    |
    v
[3.3 Handling input safely]
    |
    v
[3.4 Testing]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX007]

Learning outcomes

You will (1) write simple Python programs with variables, conditions, loops, functions and lists; (2) read the labs in this book; (3) handle input safely; and (4) test small functions.

3.1 Why programming matters for AI security

Security work needs evidence, and evidence comes from running things: evaluating a model on a test set, parsing logs, checking files. Even short programs make you faster and more precise. The labs in this book use only the Python standard library so they run on any computer.

Rina's first useful script measured how often Ask Lumen's test answers cited a clinic document, a check that would have taken hours by hand. Writing it also forced her to define what counted as a citation, which was a more precise question than the one she started with. Standard-library labs avoid dependency problems, so a classmate can rerun her work and get the same numbers.

3.2 Core constructs

Variables hold values; if chooses; for repeats; functions package steps; lists and dictionaries hold collections. Read the evasion lab below: it creates synthetic data, trains a tiny logistic-regression model with a loop, then measures accuracy as a perturbation grows. You do not need to understand every line yet — focus on the structure: data, training, evaluation, printing.

"""Lab E: evasion of a toy linear classifier (synthetic data, stdlib only).
Trains logistic regression on two synthetic clusters, then applies a
fast-gradient-sign (FGSM-style) perturbation of size eps to every test point
and reports clean vs adversarial accuracy. Teaching model only."""
import math, random
rng = random.Random(7)
def point(label):
    cx = 1.0 if label else -1.0
    return [rng.gauss(cx, 0.7), rng.gauss(cx, 0.7)], label
train = [point(i % 2) for i in range(400)]
test = [point(i % 2) for i in range(200)]
w, b = [0.0, 0.0], 0.0
sig = lambda z: 1 / (1 + math.exp(-z))
for epoch in range(200):
    gw, gb = [0.0, 0.0], 0.0
    for x, y in train:
        p = sig(w[0]*x[0] + w[1]*x[1] + b)
        gw[0] += (p - y)*x[0]; gw[1] += (p - y)*x[1]; gb += p - y
    w = [w[i] - 0.1*gw[i]/len(train) for i in range(2)]; b -= 0.1*gb/len(train)
def acc(data, eps):
    ok = 0
    for x, y in data:
        # gradient of the loss w.r.t. x is (p - y) * w; FGSM steps along its sign
        p = sig(w[0]*x[0] + w[1]*x[1] + b)
        xa = [x[i] + eps*math.copysign(1, (p - y)*w[i]) for i in range(2)]
        ok += (sig(w[0]*xa[0] + w[1]*xa[1] + b) >= 0.5) == bool(y)
    return ok / len(data)
print(f"weights w=({w[0]:.3f}, {w[1]:.3f}) b={b:.3f}")
for eps in (0.0, 0.25, 0.5, 0.75, 1.0):
    print(f"eps={eps:.2f} accuracy={acc(test, eps):.3f}")

Expected output:

weights w=(1.795, 1.756) b=-0.080
eps=0.00 accuracy=0.985
eps=0.25 accuracy=0.950
eps=0.50 accuracy=0.840
eps=0.75 accuracy=0.660
eps=1.00 accuracy=0.505

The output shows the pattern clearly: accuracy is 0.985 with no perturbation and drops to 0.505 when eps reaches one, close to guessing. The fixed random seed makes every run print the same numbers.

3.3 Handling input safely

Programs that read files, network data or AI output must treat them as untrusted. Convert text to numbers carefully, check ranges, and never execute text as code. A common LLM-application bug is passing model output into a command or web page without checks; OWASP LLM 2025 calls this improper output handling (LLM05).

For Ask Lumen, the model's suggested booking is input like any other: it must be parsed as data, checked against rules, and only then passed to the scheduling system. The worked example in this chapter does exactly that with three checks on clinic code, date and patient ID.

3.4 Testing

Write small tests with assert: assert classify(0.9) == "high". Test edge cases — empty input, extreme values, wrong types. Tests make it safe to change code later.

For Rina's booking check, edge cases include a date exactly 90 days ahead, a date in the past, an unknown clinic code, a missing patient ID and a request for another patient's ID. Each case gets one test with the expected result written beside it. When she later changes the date rule, the existing tests show at once whether anything else broke.

3.5 Worked example (synthetic LCH case)

Rina wrote a function that checks a booking request produced by Ask Lumen before it reaches the scheduling system: the clinic code must be one of five known codes, the date must be within 90 days, and the patient ID must match the logged-in user. Her tests included a request for another patient's ID, which the function rejected.

Checking in code, outside the model, means the rule holds even if a prompt persuades the model to ask for another patient's booking.

3.6 Errors and limits

Copying code without understanding what it does.
Trusting AI output as if it were your own code.
Testing only the normal case.

Copied code may include behaviour the copier never noticed, such as an unsafe call that runs text as a command. Trusting AI output as one's own code repeats that error with a less careful author. Testing only the normal case leaves exactly the paths an attacker uses untested; Rina's test with another patient's ID is the kind that catches real problems.

Exercises

EX007 — Lab structure. Identify the data, training, evaluation and output parts of the evasion lab. Rubric: 3 = all four located correctly; 2 = three; 1 = one or two; 0 = none.
EX008 — Lab reading. What accuracy does the lab report at eps 0.00 and 0.50? Rubric: 3 = 0.985 and 0.840; 2 = one; 1 = approximate; 0 = wrong.
EX009 — Validation. List the three checks in Rina's booking function and one more she could add. Rubric: 3 = clinic code, date window, patient match + sensible extra (time slot format, rate limit); 2 = three only; 1 = one or two; 0 = none.

Synthesis

Programming turns security ideas into evidence. Read labs for structure, treat every external input — including AI output — as untrusted, and test the edges.

Continue studying — chapter 03 of 40

← Chapter 02 (draft) · Course chapter index · Chapter 04 (draft) →