Chapter 14 — Software engineering
Lesson map
Lesson 14 published. Map, example, EX040.
flowchart TD
N0["Learning outcomes"]
N1["14.1 A lifecycle for AI software"]
N2["14.2 Configuration management for models and prompts"]
N3["14.3 Review and continuous integration"]
N4["14.4 Release and rollback"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX040"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["14.1 A lifecycle for AI software"]
N2["14.2 Configuration management for models and prompts"]
N3["14.3 Review and continuous integration"]
N4["14.4 Release and rollback"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX040"]
[Learning outcomes]
|
v
[14.1 A lifecycle for AI software]
|
v
[14.2 Configuration management for models and prompts]
|
v
[14.3 Review and continuous integration]
|
v
[14.4 Release and rollback]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX040]
Learning outcomes
You will (1) apply a secure development lifecycle to AI-enabled software; (2) manage requirements, versions and changes for models and prompts as well as code; (3) run code review and continuous integration with security and evaluation gates; and (4) plan releases and rollbacks for AI components.
14.1 A lifecycle for AI software
A secure development lifecycle adds security activities to each phase: requirements (abuse cases), design (threat modelling), implementation (secure coding, dependency control), verification (tests, evaluations, red-teaming) and release (sign-off, monitoring). For AI, the artefacts multiply: code, model weights, training data, prompts, retrieval indexes and evaluation sets. Each must be versioned and reviewed; a changed system prompt can alter security behaviour as much as a code change.
Abuse cases are written alongside normal requirements, for example 'a supplier bulletin tries to make Wrench create a work order'.
14.2 Configuration management for models and prompts
Store prompts in version control, not in a web console. Give each model a registry entry with its hash, training-data version, evaluation results and approval status. Deploy by reference to a registry version, so that production can always be traced and rolled back.
A prompt edited in a web console has no review, no history and no link to the evaluation that approved it. In version control, every change has an author and a diff, and the CI gate runs on it like any code change. The registry entry ties a deployed model to the data and tests behind it, so an auditor can ask what is running and get an exact answer.
14.3 Review and continuous integration
Code review catches design and logic errors; for AI components also review prompt changes and data changes. Continuous integration should run unit tests, dependency checks, and a security evaluation gate: a fixed suite of injection, jailbreak and misuse tests with thresholds that block release if exceeded. The gate must use held-out tests, not the examples developers tuned against.
Held-out tests are kept by a separate owner and refreshed with new attack patterns, so the gate measures generalisation rather than memory.
14.4 Release and rollback
Release gradually (canary deployment to a small share of traffic), watch metrics, and keep the previous version ready. For models, rollback means switching the registry reference — easy only if you planned for it.
A canary exposes a new version to a small share of real traffic while the old version serves the rest. Watched metrics include security signals, such as rejected tool proposals and detector alerts, not only latency and errors. Rollback must be practised: Sam's team switches back to the previous registry version in staging each month to confirm it still works, since an untried rollback may fail the day it is needed.
14.5 Worked example (synthetic NRS case)
A developer changed Wrench's system prompt to make answers friendlier. The CI security gate ran 300 held-out injection cases and found unauthorised-tool-call proposals rising from 1.0% to 4.3% (the policy layer still blocked them, but the gate's threshold for proposals was 2%). The release was blocked; the team found that a new sentence "always try to complete the user's goal" encouraged following instructions in retrieved text. They reworded it and the rate returned to 1.1%.
The policy layer meant no harm reached production, but the gate caught a weakening in an earlier layer before it shipped.
14.6 Errors and limits
- Prompts edited directly in production.
- Security tests that developers have seen and tuned to.
- No rollback path for models.
Production prompt edits bypass review and leave no record of why behaviour changed. Tests that developers have tuned against show the system can pass those tests, not that it resists new attacks. Models without a rollback path force teams to fix problems forward under pressure. Each gap has a direct remedy: prompts in version control, a held-out gate owned separately, and a practised switch back to the last approved registry version.
Exercises
- EX040 — Artefacts. List six artefacts of an LLM application that need version control. Rubric: 3 = code, weights/model ID, prompts, retrieval index/data, eval sets, configs/policies; 2 = four; 1 = two; 0 = none.
- EX041 — Security gate. Design a CI gate for Wrench (tests, metric, threshold, action). Rubric: 3 = held-out suite, rate metric with threshold, blocks release; 2 = missing one; 1 = vague; 0 = none.
- EX042 — Prompt change. Why did a friendliness change affect security? Rubric: 3 = prompt wording changes instruction-following priorities, including toward injected text; 2 = partial; 1 = vague; 0 = wrong.
Synthesis
AI software engineering treats models, prompts and data as versioned, reviewed artefacts, gated by held-out security evaluations and released with a rollback path.
Continue studying — chapter 14 of 50
← Chapter 13 (draft) · Course chapter index · Chapter 15 (draft) →