◇ OpenCyber Academy
Master's · Chapter 2 · English

Chapter 2 — AI and machine-learning security: advanced seminar

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 2 — AI and machine-learning security: advanced seminar

Lesson map

Lesson 02 published. Map, example, EX008.

flowchart TD
  N0["Learning outcomes"]
  N1["2.1 The seminar format"]
  N2["2.2 Evasion"]
  N3["2.3 Poisoning and backdoors"]
  N4["2.4 Privacy attacks"]
  N5["2.5 Model extraction"]
  N6["2.6 Prompt injection and LLM applications"]
  N7["2.8 Practice activities"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX008"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["2.1 The seminar format"]
  N2["2.2 Evasion"]
  N3["2.3 Poisoning and backdoors"]
  N4["2.4 Privacy attacks"]
  N5["2.5 Model extraction"]
  N6["2.6 Prompt injection and LLM applications"]
  N7["2.8 Practice activities"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX008"]

[Learning outcomes]
    |
    v
[2.1 The seminar format]
    |
    v
[2.2 Evasion]
    |
    v
[2.3 Poisoning and backdoors]
    |
    v
[2.4 Privacy attacks]
    |
    v
[2.5 Model extraction]
    |
    v
[2.6 Prompt injection and LLM applications]
    |
    v
[2.8 Practice activities]
    |
    v
[Worked example] --> [Exercise EX008]

Learning outcomes

You will (1) explain the main attack classes — evasion, poisoning and backdoors, privacy (membership inference, data extraction), model extraction, and prompt injection — with their threat models; (2) explain the defences and their known limits; (3) read and present a paper critically in a seminar; (4) relate results to NIST AI 100-2 E2025, OWASP LLM 2025 and MITRE ATLAS; and (5) identify how each attack class appears in a real deployment such as Harbourline's.

2.1 The seminar format

Each week one student presents a paper in fifteen minutes: claim, threat model, method, results, and three critical questions. The group discusses whether the evidence supports the claim. The aim is not to admire papers but to understand exactly what they show. Leo's presentation of Shokri et al. in the worked example below follows this format, and his third question became part of his own design. Prepare questions that a reader could answer from the paper, and note those it cannot answer, since those are often where research gaps lie.

2.2 Evasion

Adversarial examples (Szegedy et al., arXiv:1312.6199) are inputs changed slightly to cause errors. FGSM (Goodfellow et al., arXiv:1412.6572) takes one step along the sign of the loss gradient; PGD (Madry et al., arXiv:1706.06083) iterates and projects back into the allowed perturbation set; adversarial training uses such examples during training. In text, "small" changes are harder to define: synonym swaps or character edits may change meaning. For Harbourline, evasion would mean wording a claim note so the classifier routes it wrongly. The toy lab shows how accuracy collapses as the budget grows.

"""Lab E: evasion of a toy linear classifier (synthetic data, stdlib only).
Trains logistic regression on two synthetic clusters, then applies a
fast-gradient-sign (FGSM-style) perturbation of size eps to every test point
and reports clean vs adversarial accuracy. Teaching model only."""
import math, random
rng = random.Random(7)
def point(label):
    cx = 1.0 if label else -1.0
    return [rng.gauss(cx, 0.7), rng.gauss(cx, 0.7)], label
train = [point(i % 2) for i in range(400)]
test = [point(i % 2) for i in range(200)]
w, b = [0.0, 0.0], 0.0
sig = lambda z: 1 / (1 + math.exp(-z))
for epoch in range(200):
    gw, gb = [0.0, 0.0], 0.0
    for x, y in train:
        p = sig(w[0]*x[0] + w[1]*x[1] + b)
        gw[0] += (p - y)*x[0]; gw[1] += (p - y)*x[1]; gb += p - y
    w = [w[i] - 0.1*gw[i]/len(train) for i in range(2)]; b -= 0.1*gb/len(train)
def acc(data, eps):
    ok = 0
    for x, y in data:
        # gradient of the loss w.r.t. x is (p - y) * w; FGSM steps along its sign
        p = sig(w[0]*x[0] + w[1]*x[1] + b)
        xa = [x[i] + eps*math.copysign(1, (p - y)*w[i]) for i in range(2)]
        ok += (sig(w[0]*xa[0] + w[1]*xa[1] + b) >= 0.5) == bool(y)
    return ok / len(data)
print(f"weights w=({w[0]:.3f}, {w[1]:.3f}) b={b:.3f}")
for eps in (0.0, 0.25, 0.5, 0.75, 1.0):
    print(f"eps={eps:.2f} accuracy={acc(test, eps):.3f}")

Expected output:

weights w=(1.795, 1.756) b=-0.080
eps=0.00 accuracy=0.985
eps=0.25 accuracy=0.950
eps=0.50 accuracy=0.840
eps=0.75 accuracy=0.660
eps=1.00 accuracy=0.505

2.3 Poisoning and backdoors

Poisoning attacks control part of the training data. Availability poisoning degrades accuracy broadly; targeted and backdoor poisoning (Gu et al., arXiv:1708.06733) keep clean accuracy while a trigger causes chosen behaviour. For Harbourline, a malicious claims handler who can insert notes into the training set could teach the classifier that a phrase routes claims to fast-track approval. Defences include data provenance, filtering of outliers, and limiting who can contribute training data; none is complete. Ask, for any dataset, who can add records and whether that is logged.

2.4 Privacy attacks

Membership inference (Shokri et al., arXiv:1610.05820) decides whether a record was in training, using differences in model confidence or loss. Training-data extraction (Carlini et al., arXiv:2012.07805) recovers verbatim sequences from generative models. Differential privacy (Abadi et al., arXiv:1607.00133) bounds how much any one record can influence the model, reducing both risks at a utility cost. For an insurer, membership itself can be sensitive ("this person made a claim"). Leo's dissertation studies the first of these: whether deduplication reduces membership leakage. A claimant whose note appears many times is, on this reasoning, more exposed.

2.5 Model extraction

Model extraction (Tramèr et al., arXiv:1609.02943) builds a copy of a model from its API outputs. Rich outputs (full probability vectors) make it easier; rounding and returning only labels make it harder but not impossible. Extraction matters for intellectual property and because a stolen copy enables white-box attacks. The lab below shows the effect of output precision: with the full-precision score the copy is exact, with two decimals it is nearly exact, and with whole-number rounding the copy agrees on 88% of labels. For Harbourline, returning only the triage label would make extraction harder, not impossible.

"""Lab X: model extraction of a toy linear scoring API by querying it.
The 'API' returns a real-valued score; d+1 well-chosen queries recover the
weights exactly, and rounding the output limits fidelity. Synthetic model."""
import random
rng = random.Random(9)
d = 5
secret_w = [round(rng.uniform(-2, 2), 3) for _ in range(d)]; secret_b = 0.42
def api(x, decimals=None):
    s = sum(a*b for a, b in zip(secret_w, x)) + secret_b
    return round(s, decimals) if decimals is not None else s
for dec in (None, 2, 0):
    b = api([0]*d, dec)
    w = [api([1 if j == i else 0 for j in range(d)], dec) - b for i in range(d)]
    err = max(abs(a-c) for a, c in zip(w, secret_w))
    test = [[rng.uniform(-1, 1) for _ in range(d)] for _ in range(500)]
    agree = sum((sum(a*c for a, c in zip(w, x))+b > 0) == (api(x) > 0) for x in test)/500
    print(f"output_rounding={str(dec):4s} queries={d+1} max_weight_error={err:.3f} label_agreement={agree:.3f}")

Expected output:

output_rounding=None queries=6 max_weight_error=0.000 label_agreement=1.000
output_rounding=2    queries=6 max_weight_error=0.004 label_agreement=0.998
output_rounding=0    queries=6 max_weight_error=0.534 label_agreement=0.882

2.6 Prompt injection and LLM applications

When an LLM processes untrusted text, that text may contain instructions (Greshake et al., arXiv:2302.12173). OWASP LLM 2025 ranks prompt injection first and lists related risks such as sensitive information disclosure, excessive agency and system prompt leakage. Harbourline's future plan — an assistant that reads claim e-mails and drafts responses — would inherit these risks. For such an assistant, a claimant's e-mail is untrusted text that could steer drafts or leak other claimants' details. That plan is outside Leo's dissertation, but the seminar should map it to the OWASP entries so Harbourline can see which risks it would add.

2.7 Worked example (synthetic Harbourline case)

Leo presented Shokri et al. His three critical questions: (1) the shadow-model approach assumes the attacker can sample from a similar data distribution — is that realistic for insurance notes? (2) results were reported mostly as precision/accuracy; what is TPR at low FPR? (3) how much of the leakage is explained by overfitting? The group concluded that question (3) was directly relevant to his dissertation: if deduplication helps mainly by reducing overfitting, the "explanation" is generalisation, not deduplication per se — a competing explanation for Chapter 4.

2.8 Practice activities

  1. Present one paper using the seminar format. 2. For each attack class, write a one-line threat model for a system you know. 3. Map Harbourline's planned e-mail assistant to three OWASP LLM 2025 entries. For the presentation, prepare claim, threat model, method, results and three critical questions, as Leo did. For the threat models, name attacker knowledge, access and goal for each class. For the mapping, consider prompt injection, sensitive information disclosure and excessive agency, and say for each what the assistant could do wrong and what control would limit it.

2.9 Errors and limits

Four errors recur, and each leads to a misleading summary; check your seminar notes for all four:

Exercises

Synthesis

The seminar builds fluency in the attack classes and in critical reading. Each class has a threat model, a typical harm and defences with known limits — and the competing explanations you discover in discussion become research questions.

Continue studying — chapter 02 of 08

← Chapter 01 (draft) · Course chapter index · Chapter 03 (draft) →