OpenCyber Academy
Rapid course · Chapter 4 · English
Course code: OCA-020Chapter 04: OCA-020-C004

Chapter 4 — AI and machine-learning security: guided lab and applied synthesis

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 04 published. Map, example, EX039.

flowchart TD
  N0["Learning outcomes"]
  N1["4.1 Lab rules"]
  N2["4.2 Lab 1 — evasion"]
  N3["4.4 Lab 3 — base rates"]
  N4["4.5 Lab 4 — measuring defences"]
  N5["4.6 The adoption plan"]
  N6["4.8 Next learning"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX039"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["4.1 Lab rules"]
  N2["4.2 Lab 1 — evasion"]
  N3["4.4 Lab 3 — base rates"]
  N4["4.5 Lab 4 — measuring defences"]
  N5["4.6 The adoption plan"]
  N6["4.8 Next learning"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX039"]

[Learning outcomes]
    |
    v
[4.1 Lab rules]
    |
    v
[4.2 Lab 1 — evasion]
    |
    v
[4.4 Lab 3 — base rates]
    |
    v
[4.5 Lab 4 — measuring defences]
    |
    v
[4.6 The adoption plan]
    |
    v
[4.8 Next learning]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX039]

Learning outcomes

You will (1) run four short offline labs and interpret their outputs; (2) connect each lab to a practical control; (3) assemble a one-page adoption plan; (4) define monitoring and incident steps; and (5) plan next learning.

4.1 Lab rules

The labs run offline with Python 3 and synthetic data. They are toy models that make ideas measurable. Do not use them, or anything like them, against systems you are not authorised to test.

Read each lab's output and its lesson before moving on. Change one input at a time and predict the result before running it. If a number differs from the expected output, check your Python version and that you copied the code exactly. Keep notes of what you changed.

4.2 Lab 1 — evasion

"""Lab E: evasion of a toy linear classifier (synthetic data, stdlib only).
Trains logistic regression on two synthetic clusters, then applies a
fast-gradient-sign (FGSM-style) perturbation of size eps to every test point
and reports clean vs adversarial accuracy. Teaching model only."""
import math, random
rng = random.Random(7)
def point(label):
    cx = 1.0 if label else -1.0
    return [rng.gauss(cx, 0.7), rng.gauss(cx, 0.7)], label
train = [point(i % 2) for i in range(400)]
test = [point(i % 2) for i in range(200)]
w, b = [0.0, 0.0], 0.0
sig = lambda z: 1 / (1 + math.exp(-z))
for epoch in range(200):
    gw, gb = [0.0, 0.0], 0.0
    for x, y in train:
        p = sig(w[0]*x[0] + w[1]*x[1] + b)
        gw[0] += (p - y)*x[0]; gw[1] += (p - y)*x[1]; gb += p - y
    w = [w[i] - 0.1*gw[i]/len(train) for i in range(2)]; b -= 0.1*gb/len(train)
def acc(data, eps):
    ok = 0
    for x, y in data:
        # gradient of the loss w.r.t. x is (p - y) * w; FGSM steps along its sign
        p = sig(w[0]*x[0] + w[1]*x[1] + b)
        xa = [x[i] + eps*math.copysign(1, (p - y)*w[i]) for i in range(2)]
        ok += (sig(w[0]*xa[0] + w[1]*xa[1] + b) >= 0.5) == bool(y)
    return ok / len(data)
print(f"weights w=({w[0]:.3f}, {w[1]:.3f}) b={b:.3f}")
for eps in (0.0, 0.25, 0.5, 0.75, 1.0):
    print(f"eps={eps:.2f} accuracy={acc(test, eps):.3f}")

Expected output:

weights w=(1.795, 1.756) b=-0.080
eps=0.00 accuracy=0.985
eps=0.25 accuracy=0.950
eps=0.50 accuracy=0.840
eps=0.75 accuracy=0.660
eps=1.00 accuracy=0.505

A tiny classifier's accuracy falls from 0.985 to 0.505 as inputs are nudged in the worst direction. Lesson for ReceiptSort: do not let a single model output make important decisions without checks; review unusual or high-value items.

Run it, then change the largest epsilon and watch accuracy fall further. For ReceiptSort, an altered receipt image could be classified wrongly in the same way, which is why Omar's plan adds manual review for receipts above €500 or with low confidence. Keep notes of each run.

4.3 Lab 2 — rate limits

"""Lab R: token-bucket limits against unbounded consumption (synthetic).
Simulates 60 seconds of requests from a normal client and a flooding client
against a model endpoint with a per-client token bucket."""
import random
rng = random.Random(4)
def simulate(rate_per_s, burst, reqs):
    tokens, last, ok = burst, 0.0, 0
    for t in reqs:
        tokens = min(burst, tokens + (t - last) * rate_per_s); last = t
        if tokens >= 1: tokens -= 1; ok += 1
    return ok
normal = sorted(rng.uniform(0, 60) for _ in range(90))
flood = sorted(rng.uniform(0, 60) for _ in range(3000))
for rate, burst in ((5, 20), (2, 10), (1, 5)):
    print(f"limit={rate}/s burst={burst:2d} normal_served={simulate(rate, burst, normal):3d}/90 "
          f"flood_served={simulate(rate, burst, flood):4d}/3000")

Expected output:

limit=5/s burst=20 normal_served= 90/90 flood_served= 319/3000
limit=2/s burst=10 normal_served= 90/90 flood_served= 129/3000
limit=1/s burst= 5 normal_served= 62/90 flood_served=  64/3000

At 2 requests per second with a burst of 10, all normal requests are served while a flood is cut to 129 of 3,000. At 1 per second, normal users suffer. Lesson: set limits to protect cost and availability without blocking real work.

For InboxAide, a limit protects the firm's bill and the vendor service if an account is misused. Choose limits from normal use, as the 90-request client shows, and review them when usage grows. Note the limit you would choose and why.

4.4 Lab 3 — base rates

"""Lab B: base rates and alarm precision (Bayes' rule).
A detector for malicious prompts with given true- and false-positive rates
is applied to traffic with different shares of malicious prompts."""
for tpr, fpr in ((0.95, 0.05), (0.95, 0.01), (0.80, 0.001)):
    for prev in (0.10, 0.01, 0.001):
        tp = tpr * prev; fp = fpr * (1 - prev)
        prec = tp / (tp + fp)
        per_10k = round(10000 * (tp + fp))
        print(f"tpr={tpr:.2f} fpr={fpr:.3f} prevalence={prev:.3f} precision={prec:.3f} alerts_per_10k={per_10k}")

Expected output:

tpr=0.95 fpr=0.050 prevalence=0.100 precision=0.679 alerts_per_10k=1400
tpr=0.95 fpr=0.050 prevalence=0.010 precision=0.161 alerts_per_10k=590
tpr=0.95 fpr=0.050 prevalence=0.001 precision=0.019 alerts_per_10k=509
tpr=0.95 fpr=0.010 prevalence=0.100 precision=0.913 alerts_per_10k=1040
tpr=0.95 fpr=0.010 prevalence=0.010 precision=0.490 alerts_per_10k=194
tpr=0.95 fpr=0.010 prevalence=0.001 precision=0.087 alerts_per_10k=109
tpr=0.80 fpr=0.001 prevalence=0.100 precision=0.989 alerts_per_10k=809
tpr=0.80 fpr=0.001 prevalence=0.010 precision=0.890 alerts_per_10k=90
tpr=0.80 fpr=0.001 prevalence=0.001 precision=0.445 alerts_per_10k=18

A detector with 95% detection and 5% false alarms finds mostly false alarms when real attacks are rare (precision 1.9% at 0.1% prevalence). Lesson: choose alerts that are rare and meaningful, or staff will learn to ignore them.

For Tallis & Co, this means an alert on every suspicious prompt would mostly fire on harmless email, since injected email is rare. Omar's plan instead watches actions such as external sends and unusual volumes, which are rarer and more meaningful. Note which detector you would accept.

4.5 Lab 4 — measuring defences

The statistics lab from Chapter 2 showed how intervals shrink with more trials and how to compare a defended and undefended system (attack success 0.35 vs 0.19, reduction 12–20 points). Lesson: ask vendors and colleagues for numbers with sample sizes.

Run Lab S again from Chapter 2 and look at the last line. When a vendor says a defence works, ask how many trials were run, what the rate was with and without the defence, and how wide the interval is. A claim without these numbers cannot be compared with Omar's own results.

4.6 The adoption plan

One page: purpose; options considered; chosen option; controls (access, actions, limits, confirmation); evidence so far; monitoring (actions per user, external sends, unusual volumes); incident steps (disable feature, preserve logs, notify owner, inform affected clients if needed); owner; review triggers.

Omar's plan for InboxAide fits this list and fits on one page. It records draft-only use as the evidence so far, sends initiated only by a human click as the main control, monthly review of action logs as monitoring, and the admin switch as the first incident step. Each item names who does it. Review the plan at each trigger and keep earlier versions.

4.7 Worked example (synthetic Tallis & Co case)

After the four-week pilot — 10 users, 2,400 drafts, no sending — Omar's plan recommends wider use with sending enabled only by a human click on each message, recipient warnings for new external domains, external sharing blocked by the admin switch, and monthly review of action logs. Evidence: 0 assistant-initiated sends (sending disabled), 6/20 injected drafts in testing contained risky suggestions that staff would have had to approve. ReceiptSort: adopt with manual review for receipts above €500 or with low confidence. Incident step one: the admin switch to disable InboxAide's actions firm-wide, tested in a drill in 3 minutes.

4.8 Next learning

Follow OWASP LLM Top 10 and NIST updates; look at MITRE ATLAS case studies; consider the six-month professional course for deeper practice.

Set a date each quarter to check for new versions of the OWASP list and the NIST profiles, and note any change that affects Tallis & Co's plan. Bring one ATLAS case study to the next review as an example of what could go wrong. Share what you learn with the managing partner, so the decision record stays current. Keep a short log of what changed.

4.9 Errors and limits

Each error weakens the plan. Lab numbers come from toy models and synthetic data; they teach ideas, not real attack rates. A plan without incident steps leaves staff unsure what to do when something goes wrong, which is why Omar tested the admin switch in a drill. Without a review after adoption, vendor changes and new features go unchecked. Check your plan for all three.

Exercises

Synthesis

The labs make four ideas concrete — crafted inputs, limits, base rates and measurement — and the adoption plan ties them into a decision with controls, monitoring, incident steps and review.

Continue studying — chapter 04 of 04

← Chapter 03 (draft) · Course chapter index · Worked results ↓ · Next course via chapter index →

Worked answers

EX039. Small crafted changes make the toy model wrong about 16% of the time, up from 1.5%. So review high-value or unusual items. This is a toy result.

EX040. Send to manual review if the amount is over €500, model confidence is under 0.8, or the category is "other". High value and uncertainty are where errors cost most.

EX041. 2/s with a burst of 10. All 90 normal requests are served and the flood falls to 129/3,000. 1/s blocks normal users (62/90).

EX042. At 0.1% prevalence, about 98% of alerts are false because real events are so rare, even with 95% detection.

EX043. An external email sent to a new domain with attachments within 5 minutes of reading an outside email. More than 20 assistant actions per user per hour.

EX044. How many attacks were tested? Were they adaptive? Was success measured by effects? What is the false-block rate on normal traffic?

EX045. - Purpose: drafting help. - Options: a–d. - Chosen: c. - Controls: click-to-send, recipient warnings, sharing blocked, own-mailbox access. - Evidence: pilot and test results. - Monitoring: action logs and alerts. - Incident: admin switch, logs, notify. - Owner: managing partner. - Triggers: 3 months, incident, vendor change.

EX046. A person sees the exact action before it happens, which bounds automated harm. The remaining risk is approval fatigue, so keep confirmations meaningful.

EX047. It proves the switch works and shows how fast it is. Staff also know what to do under pressure.

EX048. Weekly review of assistant-sent emails to new external domains; per-user action volumes; admin-switch and setting changes.

EX049. A three-month review date, any incident or near miss, and any vendor feature or model change.

EX050. Answers vary. Example: "Read the OWASP LLM Top 10 entries and two ATLAS case studies by month 1. Run our injection tests again after the vendor update in month 2. Enrol in the professional course by month 3. Test only with authorisation."