Chapter 9 — AI and machine-learning security: leadership and assurance
Lesson map
Lesson 09 published. Map, example, EX041.
flowchart TD
N0["Learning outcomes"]
N1["9.1 Assurance cases"]
N2["9.2 Reporting upward"]
N3["9.3 Audit and red teaming"]
N4["9.4 Leading contested decisions"]
N5["9.5 Keeping assurance current"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX041"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["9.1 Assurance cases"]
N2["9.2 Reporting upward"]
N3["9.3 Audit and red teaming"]
N4["9.4 Leading contested decisions"]
N5["9.5 Keeping assurance current"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX041"]
[Learning outcomes]
|
v
[9.1 Assurance cases]
|
v
[9.2 Reporting upward]
|
v
[9.3 Audit and red teaming]
|
v
[9.4 Leading contested decisions]
|
v
[9.5 Keeping assurance current]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX041]
Learning outcomes
You will (1) build an assurance case for an AI system; (2) report AI-security posture to executives and boards; (3) plan internal audit and independent red teaming; (4) lead teams through contested decisions; and (5) maintain assurance as systems change.
9.1 Assurance cases
An assurance case is a structured argument: a claim ("Copilot cannot issue refunds outside policy"), arguments ("refunds pass a policy service independent of the model"), and evidence (tests, configuration, monitoring). Each claim should state its conditions and the evidence's age. Weak points are explicit: what would make the claim false?
Amira's assurance case for the refund cap lists carrier-email compromise as its weak point. For the refund cap, the evidence is the policy-service configuration, tests showing 0 of 300 adaptive cases reached payment, and monitoring of live refunds. Record the date of each piece of evidence and the model version it covers, and review the case when either changes.
9.2 Reporting upward
Boards need a short, honest picture: which AI systems carry the most risk, how much of that risk is treated, what incidents occurred, which decisions need them. Use few metrics, trends, and plain statements of residual risk. Do not hide uncertainty behind green dashboards.
Amira's one-page board report in the worked example below shows top risks with trends, incidents, evaluation coverage and one decision request. Keep the same format each quarter so the board can compare. Show trends over at least four quarters.
9.3 Audit and red teaming
Internal audit checks that processes run as designed: inventories maintained, evaluations current, changes reviewed. Independent red teams test whether controls work against motivated attackers. Ganguli et al. (arXiv:2209.07858) describe lessons from red teaming language models; scope red teams on objectives (move money, exfiltrate data) rather than on lists of prompts.
At Corvane, a red team might be asked to move money through Dispatch Copilot or extract customer addresses, rather than to try a list of prompts. Agree scope, rules and reporting with the system owner before starting.
9.4 Leading contested decisions
Product teams want speed; risk teams want evidence. Lead by making the disagreement concrete: which harm, how likely, what evidence would change minds. Offer reversible paths. When you overrule, say why in writing.
"""Lab A: spotting extraction-style abuse in a synthetic API log.
Flags clients whose query volume or input diversity is anomalous."""
import random, statistics
rng = random.Random(21)
log = []
for c in range(12):
for _ in range(rng.randint(40, 90)):
log.append((f"client{c:02d}", round(rng.gauss(0, 1), 1)))
for i in range(600): # synthetic scraper: many, grid-like inputs
log.append(("client99", round(-3 + 6*(i % 61)/60, 1)))
by = {}
for c, x in log: by.setdefault(c, []).append(x)
vol = {c: len(v) for c, v in by.items()}
med = statistics.median(vol.values()); mad = statistics.median(abs(v-med) for v in vol.values())
for c in sorted(by):
z = (vol[c]-med)/(1.4826*mad); uniq = len(set(by[c]))/len(by[c])
flag = "FLAG" if z > 3.5 else "ok"
print(f"{c} queries={vol[c]:3d} robust_z={z:6.2f} distinct_ratio={uniq:.2f} {flag}")
Expected output:
client00 queries= 50 robust_z= -0.34 distinct_ratio=0.56 ok
client01 queries= 64 robust_z= 0.45 distinct_ratio=0.52 ok
client02 queries= 81 robust_z= 1.41 distinct_ratio=0.44 ok
client03 queries= 46 robust_z= -0.56 distinct_ratio=0.52 ok
client04 queries= 52 robust_z= -0.22 distinct_ratio=0.48 ok
client05 queries= 40 robust_z= -0.90 distinct_ratio=0.57 ok
client06 queries= 42 robust_z= -0.79 distinct_ratio=0.57 ok
client07 queries= 87 robust_z= 1.74 distinct_ratio=0.48 ok
client08 queries= 56 robust_z= 0.00 distinct_ratio=0.52 ok
client09 queries= 50 robust_z= -0.34 distinct_ratio=0.62 ok
client10 queries= 68 robust_z= 0.67 distinct_ratio=0.50 ok
client11 queries= 89 robust_z= 1.85 distinct_ratio=0.42 ok
client99 queries=600 robust_z= 30.58 distinct_ratio=0.10 FLAG
At Corvane, the rebooking decision used a reversible path: design B with a value cap. Record who disagreed and on what, so the decision can be revisited when evidence changes, and invite the dissenting team to propose the evidence that would change their view. Keep it short.
9.5 Keeping assurance current
Every model update, prompt change, new tool or new data source can invalidate evidence. Tie claims to versions; re-run evaluations automatically; monitor production signals (the lab flags a client whose query volume is a robust-z outlier, a sign of possible extraction or abuse).
At Corvane, each model update or new tool triggers a re-run of Copilot's evaluation suite, and the assurance case records the versions its evidence covers. Re-run the lab's outlier check on live query logs each week and review any flagged client with the system owner.
9.6 Worked example (synthetic Corvane case)
Amira's quarterly board report has one page: top five AI risks with trend arrows, two incidents (depot rebooking, a shadow-AI data paste), evaluation coverage 4 of 5 high-tier systems, and one decision request: approve raising the refund cap to €50 based on 12,000 live refunds with 2 confirmed wrongful ones. The assurance case for the cap lists its weak point: carrier-email compromise.
The decision request is supported by live evidence and states its weak point, so the board can weigh it. The report also shows which evidence is newest.
9.7 Errors and limits
- Assurance cases without conditions or dates.
- Green dashboards hiding unknowns.
- Red teams constrained to prompt lists.
Each error makes assurance stale or misleading. A claim without conditions or dates may describe a model that no longer runs. Green dashboards hide what is unknown, such as the one high-tier system at Corvane still unevaluated. Red teams limited to prompt lists cannot find chained attacks that move money; scope them on objectives such as moving money or extracting customer addresses. Check your assurance cases and reports for all three each quarter.
Exercises
- EX041 — Assurance case. Write a claim–argument–evidence chain for the rebooking control. Rubric: 3 = claim, argument, evidence, conditions, weak point; 2 = four; 1 = two; 0 = none.
- EX042 — Board page. Draft the one-page board report outline. Rubric: 3 = risks, trends, incidents, coverage, decisions; 2 = four; 1 = two; 0 = none.
- EX043 — Red-team scope. Write objectives for a red team against Copilot. Rubric: 3 = effect-based objectives, rules of engagement, out-of-scope; 2 = two; 1 = prompt list; 0 = none.
- EX044 — Monitor lab. Interpret the outlier in the monitor lab and propose a response. Rubric: 3 = robust z far above others, investigate, rate limit; 2 = two; 1 = one; 0 = wrong.
- EX045 — Cap decision. Should the board approve €50? Use the data. Rubric: 3 = rate ≈0.017% with bound, appetite, weak point; 2 = partial; 1 = assertion; 0 = none.
Synthesis
Leadership in AI security turns evidence into assurance cases, reports risk honestly upward, uses audit and red teams for independent checks and keeps claims current as systems change.
Continue studying — chapter 09 of 12
← Chapter 08 (draft) · Course chapter index · Chapter 10 (draft) →