◇ OpenCyber Academy
Postgraduate · Chapter 5 · English

Chapter 5 — AI and machine-learning security: design alternatives

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 5 — AI and machine-learning security: design alternatives

Lesson map

Lesson 05 published. Map, example, EX021.

flowchart TD
  N0["Learning outcomes"]
  N1["5.1 Generate before choosing"]
  N2["5.2 Patterns"]
  N3["5.3 Model-side versus system-side"]
  N4["5.4 Comparing designs"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX021"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["5.1 Generate before choosing"]
  N2["5.2 Patterns"]
  N3["5.3 Model-side versus system-side"]
  N4["5.4 Comparing designs"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX021"]

[Learning outcomes]
    |
    v
[5.1 Generate before choosing]
    |
    v
[5.2 Patterns]
    |
    v
[5.3 Model-side versus system-side]
    |
    v
[5.4 Comparing designs]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX021]

Learning outcomes

You will (1) generate several architectures for an AI feature; (2) compare them on security, usefulness, cost and reversibility; (3) apply patterns such as privilege separation, plan-then-execute and structured outputs; (4) evaluate model-side versus system-side defences; and (5) present a design trade-off.

5.1 Generate before choosing

A common failure is to secure the first design proposed. Generate at least three: the minimal design (the model drafts, a human acts); the bounded agent (the model calls narrow tools with external checks); and the broad agent (the model has general tools). Then compare. Often the minimal design delivers most of the value with little risk.

Amira's rebooking comparison in the worked example below starts with three designs. For rebooking at Corvane, the minimal design keeps a human click on every change. Write each design down with its harm bound before scoring any of them, so the first idea does not anchor the comparison.

5.2 Patterns

Privilege separation: one component reads untrusted content without tool access and returns structured data; another, which never sees raw untrusted text, decides actions. Plan-then-execute: the model proposes a plan in a fixed schema; a deterministic checker validates it against policy before execution. Structured outputs: actions are typed fields, not free text, so validation is exact. Scoped credentials: tool calls run with the end user's rights, never with a service account that can reach all records.

Amira's rebooking design uses privilege separation and structured outputs.

5.3 Model-side versus system-side

Model-side defences — safety training, injection classifiers, structured queries — reduce the rate of bad outputs. System-side defences — policy checks, limits, sandboxing, confirmation — bound the harm when bad outputs occur. Wei et al. (arXiv:2307.02483) analyse why safety training fails against jailbreaks, supporting the view that model-side defences should not be the only layer. Use both, but put guarantees in the system.

At Corvane, the injection classifier is model-side and the policy service is system-side; the evaluation showed only the second reached near zero harm.

5.4 Comparing designs

Score each design on: harm bound (worst-case loss per incident), expected failure rate, user value, operating cost, implementation effort and reversibility. Present a table and a recommendation, and say which scores are estimates.

For rebooking, Amira's table would show design C with an unbounded harm, design A with low value, and design B with a cap and high value. Mark which scores are measured and which are estimates. Present the table to the product owner with the recommendation, and record the decision with its owner and revisit triggers.

5.5 Worked example (synthetic Corvane case)

For rebooking, Amira compares: (A) Copilot drafts a rebooking, an agent clicks confirm — harm bound one wrong booking per click, value moderate; (B) Copilot proposes a structured rebooking checked against the shipment owner and date rules, executed automatically for shipments under €2,000 — harm bound limited by value cap, value high; (C) Copilot has a general logistics API key — harm bound the whole shipment database. She recommends B with privilege separation: a reader model parses carrier emails into fields; the planner never sees the raw email text.

5.6 Errors and limits

Each error narrows the choice unfairly. Comparing one design with doing nothing hides cheaper safe options such as the minimal design. Prompt instructions can be overridden by injected text, so they cannot be the boundary. A service-account credential lets one injected instruction reach every record, as design C's general API key would. A design review that falls into one of these traps should be sent back to the team before approval.

Exercises

Synthesis

Good AI-security design generates alternatives and compares them openly, putting guarantees in system-side controls while using model-side defences to lower failure rates.

Continue studying — chapter 05 of 12

← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →