Chapter 38 — AI and machine-learning security: validation
Lesson map
Lesson 38 published. Map, example, EX112.
flowchart TD
N0["Learning outcomes"]
N1["38.1 Verification versus validation"]
N2["38.2 Operational validation"]
N3["38.3 Red teaming as validation"]
N4["38.4 Fitness for purpose"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX112"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["38.1 Verification versus validation"]
N2["38.2 Operational validation"]
N3["38.3 Red teaming as validation"]
N4["38.4 Fitness for purpose"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX112"]
[Learning outcomes]
|
v
[38.1 Verification versus validation]
|
v
[38.2 Operational validation]
|
v
[38.3 Red teaming as validation]
|
v
[38.4 Fitness for purpose]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX112]
Learning outcomes
You will (1) distinguish verification (built right) from validation (right thing built); (2) validate AI security with users and operators in realistic conditions; (3) run red-team exercises as validation; and (4) decide fitness for purpose with documented evidence.
38.1 Verification versus validation
Verification checks that the system meets its specified requirements. Validation checks that the requirements and the system serve stakeholders' real needs in the real environment. A system can pass every requirement and still be unsafe if the requirements missed a threat or if users behave differently than assumed.
For Wrench, verification showed that the enforcement point blocks unauthorised calls. Validation asks whether technicians can do their work with it, whether they read confirmation screens and whether the threats in the model are the ones that actually appear. Both are needed: verification without validation can approve a system nobody uses safely, and validation without verification relies on impressions.
38.2 Operational validation
Run pilots in realistic conditions: real (authorised) workloads, real users, realistic adversarial content. Observe how users interact with controls — do they read approval screens, do they work around restrictions? Measure task success and security outcomes together.
Observation reveals behaviour no requirement anticipated, such as a technician who approves proposals in batches at the end of a shift, or one who copies bulletin text into chat to save time. Measuring task success with security outcomes stops a pilot from declaring success when work got faster only because people were bypassing controls. Feedback from pilot users is collected in writing, with their permission, and fed back into requirements.
38.3 Red teaming as validation
A red team with a defined scope and authorization tries to achieve attacker goals against the integrated system, using any technique within the rules. Unlike a test suite, red teams find unanticipated paths. NIST AI 600-1 lists red-teaming among suggested actions for generative AI. Report findings with severity and fixes, and convert successful paths into regression tests.
Red teams work best when independent of the developers. Developers know the intended paths and tend to test those; an outside team starts from the attacker's goal, such as an unauthorised parts order, and works backwards through whatever the system exposes. Each finding is retested after its fix.
38.4 Fitness for purpose
The validation report states: the intended use, the conditions tested, results against stakeholder needs, red-team findings and their status, residual risks, and a recommendation (deploy, deploy with conditions, do not deploy) signed by accountable owners.
Accountable owners sign because deployment is a business decision informed by engineering evidence, not an engineering decision alone. Conditions attached to a 'deploy with conditions' outcome each have an owner and a date, and the system is checked against them before the conditions are considered met. The report also states when validation must be repeated, such as after a major model change or a new integration.
38.5 Worked example (synthetic NRS case)
Wrench passed verification. During a four-week pilot at two depots, technicians asked Wrench to "do everything from the bulletin" — a usage pattern not anticipated, which raised the share of proposals derived from untrusted text. An authorised internal red team then achieved one unauthorised parts order by chaining a search result into a supplier-allowlisted e-mail with a modified part number. Validation outcome: deploy with conditions — part numbers in e-mails must match an open work order, and training for technicians on bulletin-driven requests.
38.6 Errors and limits
- Treating verification as validation.
- Red teams without scope or with the developers grading themselves.
- Recommendations without accountable sign-off.
A system that meets its specification can still fail its users if the specification missed how they work, as the bulletin requests in the pilot showed. Red teams that report to the developers they test, or that work without written scope, produce findings that are easy to dismiss or dangerous to obtain. A recommendation nobody signs leaves the deployment decision unowned when something later goes wrong.
Exercises
- EX112 — V&V. Give an example where Wrench passes verification but fails validation. Rubric: 3 = plausible example (unanticipated usage, requirement gap); 2 = partial; 1 = vague; 0 = none.
- EX113 — Red-team finding. Turn the red-team chain into a regression test. Rubric: 3 = scenario, input, expected policy outcome, pass criterion; 2 = partial; 1 = vague; 0 = none.
- EX114 — Recommendation. Write the validation recommendation sentence for Wrench. Rubric: 3 = deploy with conditions listed, owners, review date; 2 = missing one; 1 = vague; 0 = none.
Synthesis
Validation asks whether the secured system works for real people under real pressure. Pilots and red teams find what requirements missed, and accountable owners decide fitness for purpose.
Continue studying — chapter 38 of 50
← Chapter 37 (draft) · Course chapter index · Chapter 39 (draft) →