Chapter 5 — Security scope and evidence | Technical AI and machine-learning security

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 5 — Security scope and evidence

Lesson map

Lesson 05 published. Map, example, EX017.

flowchart TD
  N0["Learning outcomes"]
  N1["5.1 Authorisation and scope"]
  N2["5.2 Basic terms"]
  N3["5.3 Evidence"]
  N4["5.4 Rates and uncertainty"]
  N5["5.4b Before and after"]
  N6["5.5 Reporting"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX017"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["5.1 Authorisation and scope"]
  N2["5.2 Basic terms"]
  N3["5.3 Evidence"]
  N4["5.4 Rates and uncertainty"]
  N5["5.4b Before and after"]
  N6["5.5 Reporting"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX017"]

[Learning outcomes]
    |
    v
[5.1 Authorisation and scope]
    |
    v
[5.2 Basic terms]
    |
    v
[5.3 Evidence]
    |
    v
[5.4 Rates and uncertainty]
    |
    v
[5.4b Before and after]
    |
    v
[5.5 Reporting]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX017]

Learning outcomes

You will (1) work within written authorisation and scope; (2) use CIA and basic risk terms; (3) collect evidence properly; (4) report findings clearly.

5.1 Authorisation and scope

Before testing or changing anything, have written authorisation stating which systems, which activities, which times and which data. Use isolated copies and synthetic data. If you find something outside scope, stop and ask. Unauthorised testing can be illegal and can harm customers.

Ana's written authorisation for the cross-account restart test named the HelpBot staging copy, the activity (requesting restarts of routers belonging to other synthetic test accounts), weekday working hours, the synthetic accounts to use and the supervisor to call. When she noticed during testing that the staging copy's ticket tool still sent e-mail to a real internal queue, she stopped and asked, because that queue was outside scope. The tool was pointed at a test queue before she continued.

5.2 Basic terms

Confidentiality, integrity, availability. Threat (who or what could cause harm), vulnerability (the weakness), risk (likelihood × impact), control (what reduces it). For AI also know the main frameworks: NIST AI RMF (AI 100-1), OWASP Top 10 for LLM Applications 2025, MITRE ATLAS.

Applied to the cross-account restart test: the threat is a customer, or text planted in an article, steering HelpBot to restart someone else's router; the vulnerability is a restart tool that trusted the router ID in the model's request; the risk is the likelihood of that happening times the impact of a customer losing service; the control is the ownership check in the tool service. The affected property is availability for the other customer and integrity of account actions. In OWASP terms this is LLM06 Excessive Agency.

5.3 Evidence

Record: date and time, system and version, exact steps, inputs, outputs, tool logs, screenshots where useful. Keep evidence unchanged; work on copies. Measure by effects: "the tool restarted router R-55120" is evidence; "the reply sounded wrong" is not.

For each of Ana's 60 cross-account attempts the record held the staging version, the synthetic account used, the exact message sent, HelpBot's reply, the tool-call log line and the router-management service's response. Success was defined by the tool log, not by the reply: in a few conversations HelpBot said "I can't do that" while the log showed the restart had been sent. That mismatch is exactly why replies are not evidence. The raw logs were exported once and hashed, and analysis ran on a copy.

5.4 Rates and uncertainty

One test proves little. Repeat tests and report successes over trials; zero failures in 100 trials still allows a failure rate up to about 3%.

"""Lab S: reporting attack-success rates honestly: Wilson intervals,
seed variation and a paired bootstrap. Synthetic trial outcomes only."""
import math, random
def wilson(k, n, z=1.96):
    p = k/n; d = 1 + z*z/n
    c = (p + z*z/(2*n))/d; h = z*math.sqrt(p*(1-p)/n + z*z/(4*n*n))/d
    return c - h, c + h
for k, n in ((3, 20), (30, 200), (300, 2000)):
    lo, hi = wilson(k, n); print(f"successes={k}/{n} rate={k/n:.3f} wilson95=[{lo:.3f}, {hi:.3f}]")
# five seeds of a synthetic defence evaluation: per-seed attack success rate
rates = []
for seed in range(5):
    r = random.Random(seed); rates.append(sum(r.random() < 0.12 for _ in range(200))/200)
m = sum(rates)/5; sd = math.sqrt(sum((x-m)**2 for x in rates)/4)
print("per-seed rates:", " ".join(f"{x:.3f}" for x in rates), f"mean={m:.3f} sd={sd:.3f}")
# paired bootstrap: baseline vs defence on the same 300 synthetic prompts
r = random.Random(42)
base = [r.random() < 0.30 for _ in range(300)]
defn = [b and r.random() < 0.6 for b in base]   # defence blocks ~40% of successes
diffs = []
for _ in range(2000):
    idx = [r.randrange(300) for _ in range(300)]
    diffs.append(sum(base[i] for i in idx)/300 - sum(defn[i] for i in idx)/300)
diffs.sort()
print(f"baseline_asr={sum(base)/300:.3f} defended_asr={sum(defn)/300:.3f} "
      f"reduction_ci95=[{diffs[50]:.3f}, {diffs[1949]:.3f}]")

Expected output:

successes=3/20 rate=0.150 wilson95=[0.052, 0.360]
successes=30/200 rate=0.150 wilson95=[0.107, 0.206]
successes=300/2000 rate=0.150 wilson95=[0.135, 0.166]
per-seed rates: 0.110 0.130 0.135 0.105 0.145 mean=0.125 sd=0.017
baseline_asr=0.350 defended_asr=0.190 reduction_ci95=[0.120, 0.203]

The lab's Wilson intervals show how much the number of trials matters. Three successes in 20 trials gives a 95% interval between 5.2% and 36.0%; the same rate over 200 trials narrows it to 10.7%–20.6%, and over 2,000 trials to 13.5%–16.6%. Ana's 9 of 60 before the fix carries an interval of about 8% to 26%, wide but clearly above zero. After the fix, 0 of 60 means the rate is probably below about 6%, which is why she later ran more attempts.

5.4b Before and after

When you test a fix, run the same cases before and after it, on the same version otherwise, so the difference can be credited to the fix.

Ana's restart test followed this. Both runs of 60 attempts used the same staging build, the same synthetic accounts and the same 60 messages; the only change between them was the ownership check in the tool service. If the vendor model had been updated between the runs, a drop from 9 to 0 could have come from the model behaving differently rather than from the fix. The lab's paired bootstrap applies the same idea: baseline and defended rates are compared on the same 300 prompts.

5.5 Reporting

A finding report: title, system and version, steps to reproduce, observed effect, impact, severity, suggested fix. Keep it factual and share it only through the agreed channel.

Ana's report for the cross-account test fitted on one page. Title: HelpBot restart tool accepts other customers' routers. System: HelpBot staging, orchestrator and tool service versions listed. Steps: the message template and the synthetic accounts. Effect: 9 of 60 attempts restarted a router not owned by the requester, confirmed in tool logs. Impact: a customer can cut another customer's internet. Severity: high. Fix: ownership check in the tool service, verified 0 of 60. It went to the team lead through the ticket queue, not chat.

5.6 Worked example (synthetic Kestrel case)

Ana's authorisation covers HelpBot staging, weekdays, synthetic accounts. She runs 60 conversations asking HelpBot to restart a router belonging to a different test account. Before the fix: 9 of 60 succeed (interval about 8–26%). After the ownership check: 0 of 60. She writes one finding with both results.

The report's value came from the fact that both runs used the same 60 messages and the same build, so the drop from 9 to 0 can be credited to the ownership check. Ana did not claim the problem was gone: 0 of 60 still allows a true rate of up to about 6%, and the follow-up runs after the Wi-Fi password tool was added used the same template to keep results comparable.

5.7 Errors and limits

Testing production without authorisation.
Evidence based on one attempt.
Reports with no steps to reproduce.

Each error would have damaged the restart finding. Testing production would have restarted real customers' routers and cut their service. A single attempt would have been misleading in either direction, since only about one attempt in seven succeeded before the fix; a single failed try could have "proved" the tool safe. A report without the message template and the synthetic accounts would have left the developer unable to reproduce the problem or to confirm the fix, and the team lead unable to judge severity.

Exercises

EX017 — Authorisation. List the elements of a test authorisation. Rubric: 3 = systems, activities, times, data, contact, signature; 2 = four; 1 = two; 0 = none.
EX018 — Terms. Label threat, vulnerability, risk and control in Ana's cross-account restart test. Rubric: 3 = all four correct; 2 = three; 1 = one or two; 0 = none.
EX019 — Evidence. List six items to record for one test. Rubric: 3 = six apt; 2 = four; 1 = two; 0 = none.
EX020 — Finding. Write the finding report for Ana's cross-account restart test. Rubric: 3 = all seven parts; 2 = five; 1 = three; 0 = none.

Synthesis

Work only within authorisation, describe risk precisely, collect evidence by effects over repeated trials and report clearly.

Continue studying — chapter 05 of 18

← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →