Chapter 39 — Capstone evaluation | Bachelor's AI and machine-learning security

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 39 — Capstone evaluation

Lesson map

Lesson 39 published. Map, example, EX115.

flowchart TD
  N0["Learning outcomes"]
  N1["39.1 Final evaluation plan"]
  N2["39.2 Independent testing"]
  N3["39.3 Usability and side effects"]
  N4["39.4 Residual risk and recommendations"]
  N5["39.4b Reporting the evaluation"]
  N6["39.6b Agreeing the verdict criteria in advance"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX115"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["39.1 Final evaluation plan"]
  N2["39.2 Independent testing"]
  N3["39.3 Usability and side effects"]
  N4["39.4 Residual risk and recommendations"]
  N5["39.4b Reporting the evaluation"]
  N6["39.6b Agreeing the verdict criteria in advance"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX115"]

[Learning outcomes]
    |
    v
[39.1 Final evaluation plan]
    |
    v
[39.2 Independent testing]
    |
    v
[39.3 Usability and side effects]
    |
    v
[39.4 Residual risk and recommendations]
    |
    v
[39.4b Reporting the evaluation]
    |
    v
[39.6b Agreeing the verdict criteria in advance]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX115]

Learning outcomes

You will (1) evaluate the capstone against its acceptance criteria; (2) include independent and adaptive testing; (3) assess usability and side effects; and (4) state residual risks and recommendations.

39.1 Final evaluation plan

Use the acceptance criteria from the proposal, unchanged unless the change log records why. Run the full held-out suites, the independent adaptive suite and normal-use tests. Compute rates with intervals and compare with criteria.

Rina's final plan used the criteria from her proposal: zero cross-patient bookings with an upper bound under 0.5%, completed bookings of at least 93%, and the false-statement threshold. The only recorded change was the dropped voice channel, approved earlier. Running the full suites once, on a frozen configuration, avoided the temptation to rerun until a better result appeared.

39.2 Independent testing

Ask people who did not build the system to attack it within scope. Their results go into the evaluation even if they are unfavourable.

Rina's two classmates had already written adaptive cases during the seminar; for the final evaluation, a third tester from another course attacked the system for two days. That tester found the long-conversation quota problem, which nobody involved in building had noticed because their test conversations were short. Fresh testers bring different habits, and those habits find different failures.

39.3 Usability and side effects

Measure task completion, time, false refusals and user feedback. Security controls that users bypass or abandon do not protect anyone.

Rina measured completed bookings, average time to book and false refusals, and asked ten classmates acting as patients for brief feedback. The quota problem appeared in that feedback as frustration, not as a security failure. A booking assistant that loses context in long conversations would push patients back to the phone, which undoes the purpose of the service and adds load at reception.

39.4 Residual risk and recommendations

List risks that remain, why they are acceptable or not, and what further work would reduce them. Recommend whether and how the approach should be adopted.

"""Lab S: reporting attack-success rates honestly: Wilson intervals,
seed variation and a paired bootstrap. Synthetic trial outcomes only."""
import math, random
def wilson(k, n, z=1.96):
    p = k/n; d = 1 + z*z/n
    c = (p + z*z/(2*n))/d; h = z*math.sqrt(p*(1-p)/n + z*z/(4*n*n))/d
    return c - h, c + h
for k, n in ((3, 20), (30, 200), (300, 2000)):
    lo, hi = wilson(k, n); print(f"successes={k}/{n} rate={k/n:.3f} wilson95=[{lo:.3f}, {hi:.3f}]")
# five seeds of a synthetic defence evaluation: per-seed attack success rate
rates = []
for seed in range(5):
    r = random.Random(seed); rates.append(sum(r.random() < 0.12 for _ in range(200))/200)
m = sum(rates)/5; sd = math.sqrt(sum((x-m)**2 for x in rates)/4)
print("per-seed rates:", " ".join(f"{x:.3f}" for x in rates), f"mean={m:.3f} sd={sd:.3f}")
# paired bootstrap: baseline vs defence on the same 300 synthetic prompts
r = random.Random(42)
base = [r.random() < 0.30 for _ in range(300)]
defn = [b and r.random() < 0.6 for b in base]   # defence blocks ~40% of successes
diffs = []
for _ in range(2000):
    idx = [r.randrange(300) for _ in range(300)]
    diffs.append(sum(base[i] for i in idx)/300 - sum(defn[i] for i in idx)/300)
diffs.sort()
print(f"baseline_asr={sum(base)/300:.3f} defended_asr={sum(defn)/300:.3f} "
      f"reduction_ci95=[{diffs[50]:.3f}, {diffs[1949]:.3f}]")

Expected output:

successes=3/20 rate=0.150 wilson95=[0.052, 0.360]
successes=30/200 rate=0.150 wilson95=[0.107, 0.206]
successes=300/2000 rate=0.150 wilson95=[0.135, 0.166]
per-seed rates: 0.110 0.130 0.135 0.105 0.145 mean=0.125 sd=0.017
baseline_asr=0.350 defended_asr=0.190 reduction_ci95=[0.120, 0.203]

The lab shows why Rina reported intervals rather than single rates: identical rates of 0.150 carry very different certainty at 20 and at 2000 trials. Her residual-risk list named non-English injection and the reliance on partner-upload review, explained why each was acceptable for a pilot, and proposed tests that would reduce them before wider adoption.

39.4b Reporting the evaluation

Present one table per criterion, showing result, interval, threshold and pass or fail. Then present the independent findings, usability measures and a short discussion. Attach the evaluation scripts and test-suite descriptions so a reader could repeat the run. Report a failed criterion as a failure, with its cause and a plan, rather than redefining it.

Rina's table for completed bookings showed 94% against the 93% threshold, with its interval. Because the margin was narrow, she reported the pass with that caveat rather than presenting it as comfortable, and suggested LCH watch the rate in the pilot. The quota finding followed in the independent-testing part of the report.

39.5 Worked example (synthetic LCH case)

Rina's final evaluation: 0 cross-patient bookings in 1,000 attempts (upper bound ≈ 0.4%); misleading confirmations 0 in 300 adaptive cases; false clinic statements 2.4% (interval 1.6–3.5%), below the 4% criterion; completed bookings 94%. Independent testers found one issue: very long conversations hit the token quota and lost context, frustrating users. Residual risks: non-English injection untested; reliance on partner-upload review. Recommendation: adopt policy service and structured confirmation at LCH; extend tests to other languages.

The quota issue was reported as a usability finding with a proposed fix, raising the quota for authenticated patients.

39.6 Errors and limits

Changing criteria after results.
Leaving out unfavourable independent results.
Ignoring usability.

Changing the false-statement threshold after seeing results would have made the pass meaningless, so Rina kept the original. Leaving out the quota problem because it came from an independent tester would hide the main usability weakness. Ignoring usability would have produced a system that passed security tests while patients gave up on it and phoned the clinic instead. Reporting all three, including the unfavourable ones, keeps the evaluation honest and useful to LCH.

39.6b Agreeing the verdict criteria in advance

The capstone's pass criteria — for example, injection success below an agreed rate on a held-out attack set, with no increase in wrong bookings — are fixed before the final evaluation. If results fall between thresholds, the report says so plainly instead of rounding toward success.

Rina's criteria were signed by her supervisor before the final run, so they could not drift. Signing them first meant that when completed bookings landed close to the threshold, nobody could argue about what counted as a pass. The examiner checked the signed date against the date of the final run.

Exercises

EX115 — Criteria. Check Rina's results against her criteria and say which passed. Rubric: 3 = all stated criteria assessed correctly; 2 = most; 1 = vague; 0 = none.
EX116 — Usability issue. How should the quota issue be handled? Rubric: 3 = report it, adjust quota/summarise context, retest, balance against cost risk; 2 = partial; 1 = ignore; 0 = none.
EX117 — Residual risk. Write one residual-risk statement with justification and next step. Rubric: 3 = risk, why acceptable/not, next step; 2 = two; 1 = vague; 0 = none.

Synthesis

Capstone evaluation is honest measurement against pre-set criteria, including independent attacks and usability, ending with clear residual risks and recommendations.

Continue studying — chapter 39 of 40

← Chapter 38 (draft) · Course chapter index · Chapter 40 (draft) →