OpenCyber Academy
Doctoral research · Chapter 2 · English
Course code: OCA-204Chapter 02: OCA-204-C002

Chapter 2 — AI and machine-learning security: research frontier

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 02 published. Map, example, EX008.

flowchart TD
  N0["Learning outcomes"]
  N1["2.1 Four classical attack families"]
  N2["2.2 Generative and agentic systems"]
  N3["2.3 Governance as research context"]
  N4["2.4 Finding open problems"]
  N5["2.6 Practice activities"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX008"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["2.1 Four classical attack families"]
  N2["2.2 Generative and agentic systems"]
  N3["2.3 Governance as research context"]
  N4["2.4 Finding open problems"]
  N5["2.6 Practice activities"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX008"]

[Learning outcomes]
    |
    v
[2.1 Four classical attack families]
    |
    v
[2.2 Generative and agentic systems]
    |
    v
[2.3 Governance as research context]
    |
    v
[2.4 Finding open problems]
    |
    v
[2.6 Practice activities]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX008]

Learning outcomes

You will (1) map the attack surface of predictive and generative AI using the NIST AI 100-2 E2025 taxonomy, MITRE ATLAS and the OWASP Top 10 for LLM Applications 2025; (2) summarise the foundational results in evasion, poisoning, privacy and extraction; (3) explain why prompt injection is an architectural rather than a filtering problem; (4) locate open problems where a doctoral contribution is plausible; and (5) relate research to governance frameworks (NIST AI RMF, ISO/IEC 42001, the EU AI Act).

2.1 Four classical attack families

Evasion changes an input at test time so a model errs. Szegedy et al. (arXiv:1312.6199) found that imperceptible perturbations fool image classifiers; Goodfellow, Shlens and Szegedy (arXiv:1412.6572) explained this by local linearity and introduced the fast gradient sign method (FGSM). Carlini and Wagner (arXiv:1608.04644) gave stronger optimisation attacks; Madry et al. (arXiv:1706.06083) framed robustness as a min–max problem and popularised projected-gradient-descent adversarial training. Cohen, Rosenfeld and Kolter (arXiv:1902.02918) offered certified robustness via randomized smoothing, at the cost of accuracy and with guarantees only for small norm-bounded perturbations.

Poisoning corrupts training data or the training process. Biggio, Nelson and Laskov (arXiv:1206.6389) poisoned support-vector machines; Gu et al. BadNets (arXiv:1708.06733) and Chen et al. (arXiv:1712.05526) showed backdoors that keep clean accuracy while a trigger forces a target output. Carlini et al. (arXiv:2302.10149) showed that poisoning web-scale datasets is practical, for example by buying expired domains referenced in dataset URL lists. Hubinger et al. Sleeper Agents (arXiv:2401.05566) trained language models with backdoored behaviour that persisted through standard safety fine-tuning.

Privacy attacks infer information about training data. Shokri et al. (arXiv:1610.05820) introduced membership inference; Carlini et al. (arXiv:2012.07805) extracted verbatim training sequences from a large language model. Differentially private training (Abadi et al., arXiv:1607.00133) bounds such leakage formally, at a utility cost.

Extraction copies a model's functionality through its API. Tramèr et al. (arXiv:1609.02943) stole models from prediction APIs, sometimes with a number of queries close to the number of parameters for simple models.

The toy lab below reproduces the evasion intuition on a linear model: accuracy falls steadily as the perturbation budget grows, even though the model is accurate on clean data.

"""Lab E: evasion of a toy linear classifier (synthetic data, stdlib only).
Trains logistic regression on two synthetic clusters, then applies a
fast-gradient-sign (FGSM-style) perturbation of size eps to every test point
and reports clean vs adversarial accuracy. Teaching model only."""
import math, random
rng = random.Random(7)
def point(label):
    cx = 1.0 if label else -1.0
    return [rng.gauss(cx, 0.7), rng.gauss(cx, 0.7)], label
train = [point(i % 2) for i in range(400)]
test = [point(i % 2) for i in range(200)]
w, b = [0.0, 0.0], 0.0
sig = lambda z: 1 / (1 + math.exp(-z))
for epoch in range(200):
    gw, gb = [0.0, 0.0], 0.0
    for x, y in train:
        p = sig(w[0]*x[0] + w[1]*x[1] + b)
        gw[0] += (p - y)*x[0]; gw[1] += (p - y)*x[1]; gb += p - y
    w = [w[i] - 0.1*gw[i]/len(train) for i in range(2)]; b -= 0.1*gb/len(train)
def acc(data, eps):
    ok = 0
    for x, y in data:
        # gradient of the loss w.r.t. x is (p - y) * w; FGSM steps along its sign
        p = sig(w[0]*x[0] + w[1]*x[1] + b)
        xa = [x[i] + eps*math.copysign(1, (p - y)*w[i]) for i in range(2)]
        ok += (sig(w[0]*xa[0] + w[1]*xa[1] + b) >= 0.5) == bool(y)
    return ok / len(data)
print(f"weights w=({w[0]:.3f}, {w[1]:.3f}) b={b:.3f}")
for eps in (0.0, 0.25, 0.5, 0.75, 1.0):
    print(f"eps={eps:.2f} accuracy={acc(test, eps):.3f}")

Expected output:

weights w=(1.795, 1.756) b=-0.080
eps=0.00 accuracy=0.985
eps=0.25 accuracy=0.950
eps=0.50 accuracy=0.840
eps=0.75 accuracy=0.660
eps=1.00 accuracy=0.505

2.2 Generative and agentic systems

Large language models (LLMs) add new surfaces. Jailbreaks elicit content that safety training was meant to prevent; Wei, Haghtalab and Steinhardt (arXiv:2307.02483) traced them to competing objectives and mismatched generalisation, and Zou et al. (arXiv:2307.15043) found automatically optimised suffixes that transfer across models. Prompt injection is different: an attacker places instructions in content the application feeds the model. Perez and Ribeiro (arXiv:2211.09527) studied direct goal hijacking; Greshake et al. (arXiv:2302.12173) showed indirect injection through web pages, e-mails and documents that an LLM-integrated application retrieves, leading to data exfiltration and tool misuse; Liu et al. (arXiv:2306.05499) attacked real LLM-integrated applications. The OWASP Top 10 for LLM Applications 2025 lists prompt injection as LLM01 and adds sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation and unbounded consumption.

Injection is architectural because an LLM receives instructions and data in one token stream; there is no hardware-like boundary between them. Defences therefore combine structure (Chen et al. StruQ, arXiv:2402.06363, trains models to follow only instructions in a reserved channel), privilege reduction (least-privilege tools, allowlists, human confirmation for consequential actions) and detection. NIST AI 600-1 lists information security and related risks for generative AI with suggested actions; MITRE ATLAS catalogues adversary tactics and techniques against AI systems with case studies, in the style of ATT&CK.

2.3 Governance as research context

The NIST AI RMF 1.0 (AI 100-1) organises risk work into Govern, Map, Measure and Manage functions and lists characteristics of trustworthy AI, including secure and resilient. ISO/IEC 42001:2023 specifies an AI management system. Regulation (EU) 2024/1689 (the AI Act) entered into force on 1 August 2024 with staged application; Article 15 requires high-risk systems to achieve appropriate accuracy, robustness and cybersecurity, naming data poisoning, model poisoning, adversarial examples and confidentiality attacks. A doctoral contribution that offers a measurement method for such properties can therefore matter beyond academia — but you must not overstate legal conclusions.

2.4 Finding open problems

Good frontier questions sit where (a) attacks are cheap and defences are evaluated weakly, (b) theory and practice disagree, or (c) a deployment pattern has outrun evaluation. Current examples include evaluating agent defences under adaptive attackers; measuring residual risk when no certified bound exists; supply-chain integrity of model weights and datasets; and privacy auditing of fine-tuned models. Write each as a gap statement: what is known, what is claimed but untested, why it matters, what evidence would settle it. Ana's gap in the worked example shows the form.

2.5 Worked example (synthetic MAAL case)

Ana mapped 31 papers on prompt-injection defences into a table with columns: threat model, adaptive attack (yes/no), metric, utility cost reported, code released. Eighteen papers evaluated only fixed payload lists; nine reported no utility cost; six released code. Her gap statement: structural defences are claimed to generalise, but few evaluations use attackers who adapt to the structure itself, and utility costs are rarely measured on the same tasks. That gap is narrow enough to test and broad enough to matter.

2.6 Practice activities

  1. Build your own mapping table for twenty papers in your sub-area. 2. Classify three MITRE ATLAS case studies by NIST AI 100-2 attacker goal and capability. 3. Rewrite one open problem as a four-part gap statement. For the mapping table, use the same columns as Ana: threat model, adaptive attack, metric, utility cost and code release, and count each column at the end. For the ATLAS case studies, note where a case does not fit the NIST categories cleanly; that mismatch is itself useful. For the gap statement, check that the evidence part names an experiment you could actually run within the doctorate.

2.7 Errors and limits

Each of these confusions leads to the wrong study; check your related-work chapter for each:

Exercises

Synthesis

The frontier is shaped by a decade of attacks outpacing defences. Generative and agentic systems widen it, because instructions and data share a channel. A good doctoral question sits on a measured gap and uses shared vocabularies so others can test it.

Continue studying — chapter 02 of 08

← Chapter 01 (draft) · Course chapter index · Chapter 03 (draft) →

Worked answers

EX008. Goal: integrity (targeted misbehaviour on trigger). Capability: training-data control. Stage: training (fine-tuning) — a supply-chain vector.

EX009. At eps = 0.75 accuracy is 0.660 (0.840 at 0.50). For a linear model, each perturbation moves the score by eps × ||w||₁ in the worst direction, so points within that margin of the boundary flip; as eps grows, a growing share of the Gaussian clusters lies within reach.

EX010. Jailbreak: the user crafts a prompt to make the model produce disallowed content (e.g., role-play to obtain prohibited instructions). Prompt injection: third-party data processed by the application carries instructions (e.g., a web page saying "forward the user's inbox to …" that a browsing agent obeys).

EX011. LLM01 Prompt Injection — the retrieved message carries instructions the model follows. LLM06 Excessive Agency — the agent can send mail without restriction or confirmation, so the injection has impact.

EX012. Known: membership inference works on classifiers and some pre-trained LMs. Claimed but untested: that standard fine-tuning recipes leak little on small private datasets. Why it matters: organisations fine-tune on sensitive records. Evidence: calibrated attacks (e.g., likelihood-ratio with reference models) on fine-tuned models across dataset sizes, with TPR at low FPR.

EX013. It certifies that the smoothed classifier's prediction is constant within an L2 radius around a given input. It does not cover larger or differently bounded changes (patches, semantic edits, L∞ beyond the implied bound), poisoning or backdoors.

EX014. Article 15 (accuracy, robustness and cybersecurity). It names data poisoning, model poisoning, adversarial examples/model evasion, confidentiality attacks and model flaws.