Chapter 30 — Capstone assessment
Lesson map
Lesson 30 published. Map, example, EX088.
flowchart TD
N0["Learning outcomes"]
N1["30.1 Final evaluation"]
N2["30.2 Demonstration and defence"]
N3["30.3 Residual risk and recommendations"]
N4["30.4 Continuing development"]
N5["30.4b Assessment criteria"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX088"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["30.1 Final evaluation"]
N2["30.2 Demonstration and defence"]
N3["30.3 Residual risk and recommendations"]
N4["30.4 Continuing development"]
N5["30.4b Assessment criteria"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX088"]
[Learning outcomes]
|
v
[30.1 Final evaluation]
|
v
[30.2 Demonstration and defence]
|
v
[30.3 Residual risk and recommendations]
|
v
[30.4 Continuing development]
|
v
[30.4b Assessment criteria]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX088]
Learning outcomes
You will (1) evaluate your capstone against its requirements; (2) present and defend it; (3) state residual risks and recommendations; and (4) plan continued professional development.
30.1 Final evaluation
Run all suites on the final version: held-out, adaptive (from independent testers) and normal use. Report each requirement with result, interval and pass or fail. Include side effects and costs. Report failures honestly with cause and plan; an honest failed criterion scores better than a hidden one. Jonas's final evaluation of Gift Card Help reports cross-user disclosures, out-of-policy transfers and normal completion, each with counts. It states the transfer-volume alert's precision honestly and explains the dashboard routing. Each result shows its interval and whether it met the agreed criterion, so a failed criterion is as visible as a passed one.
30.2 Demonstration and defence
Demonstrate a normal task, an attack blocked by your controls, an alert firing and its runbook, and a rollback. Expect questions on threat model, test independence, statistics, and what would happen if the model changed. "The policy service enforces it regardless of the model" is a strong answer when your evidence supports it. Jonas's demonstration of Gift Card Help showed a balance lookup, a blocked transfer to a card the customer did not own, the transfer-volume dashboard, and a prompt rollback. Asked what would happen if the vendor model changed, he pointed to the policy service and the gate.
30.3 Residual risk and recommendations
State untested areas, accepted risks and their owners, and recommendations for production use. For Gift Card Help, Jonas states that transfers by phone support bypass the assistant's checks and names the support team's manager as the owner of that risk. He recommends applying the same same-owner rule in the phone process before production use, and reruns of the adaptive suite after any model update. He also lists what he did not test, such as other languages, so the next owner knows where evidence stops.
30.4 Continuing development
Plan: authorised practice (labs, capture-the-flag events with AI tracks), following OWASP, NIST and ATLAS updates, reading a paper a month, and contributing to open-source evaluation tools. Keep ethical commitments: authorised testing and responsible disclosure.
"""Lab B: base rates and alarm precision (Bayes' rule).
A detector for malicious prompts with given true- and false-positive rates
is applied to traffic with different shares of malicious prompts."""
for tpr, fpr in ((0.95, 0.05), (0.95, 0.01), (0.80, 0.001)):
for prev in (0.10, 0.01, 0.001):
tp = tpr * prev; fp = fpr * (1 - prev)
prec = tp / (tp + fp)
per_10k = round(10000 * (tp + fp))
print(f"tpr={tpr:.2f} fpr={fpr:.3f} prevalence={prev:.3f} precision={prec:.3f} alerts_per_10k={per_10k}")
Expected output:
tpr=0.95 fpr=0.050 prevalence=0.100 precision=0.679 alerts_per_10k=1400
tpr=0.95 fpr=0.050 prevalence=0.010 precision=0.161 alerts_per_10k=590
tpr=0.95 fpr=0.050 prevalence=0.001 precision=0.019 alerts_per_10k=509
tpr=0.95 fpr=0.010 prevalence=0.100 precision=0.913 alerts_per_10k=1040
tpr=0.95 fpr=0.010 prevalence=0.010 precision=0.490 alerts_per_10k=194
tpr=0.95 fpr=0.010 prevalence=0.001 precision=0.087 alerts_per_10k=109
tpr=0.80 fpr=0.001 prevalence=0.100 precision=0.989 alerts_per_10k=809
tpr=0.80 fpr=0.001 prevalence=0.010 precision=0.890 alerts_per_10k=90
tpr=0.80 fpr=0.001 prevalence=0.001 precision=0.445 alerts_per_10k=18
Jonas's plan names specific steps: one capture-the-flag event with an AI track per term, a monthly paper from the reading list, and a contribution of his harness's case format to an open-source evaluation project. Each is measurable, unlike a general wish to keep learning.
30.4b Assessment criteria
Assessors look for: requirements traced to evidence; controls that do not depend on model behaviour; independent adaptive testing; honest reporting of failures; operational readiness (alerts, runbooks, rollback); and clear communication. Check your submission against each before handing it in. Jonas checked Gift Card Help against each item: requirements traced to tests, the policy service independent of the model, an independent adaptive tester, the alert precision stated honestly, rehearsed rollback and a one-page summary for owners. The completed checklist is part of his submission, with each item pointing to the Gift Card Help evidence behind it.
30.5 Worked example (synthetic Brightwell case)
Jonas's final results for Gift Card Help: cross-user balance disclosures 0/300 (upper bound ≈ 1%); out-of-policy transfers 0/150 adaptive; normal-task completion 95% of 400; transfer-volume alert precision on a week of replayed traffic 0.5 — he explains with the base-rate lab why more than half of alerts on rare events can be false, and routes the alert to a dashboard rather than paging. Residual risk: phone-support channel can transfer balances without the same checks. The final results show the habits the book asks for: bounded rates, adaptive testing, honest alert precision and named residual risk.
30.6 Errors and limits
- Hiding failed criteria.
- Demonstrations without an attack.
- Development plans with no specifics.
Jonas's final submission avoids these. It reports the weak alert precision openly, demonstrates a blocked attack, and gives a development plan with named events and dates. Hiding the alert's precision would have scored worse than explaining it, and a demonstration without an attack shows only that the feature works for friendly users. Each item in his plan has a date, so his supervisor can check progress at the next review instead of relying on a general intention to keep learning.
Exercises
- EX088 — Results table. Build the results table for Jonas's capstone. Rubric: 3 = requirement, result, interval, criterion, pass/fail; 2 = four; 1 = two; 0 = none.
- EX089 — Defence question. Answer: "If the vendor updates the model, is your system still safe?" Rubric: 3 = enforcement independent of model, gate rerun on update, pinned version; 2 = two; 1 = yes/no; 0 = none.
- EX090 — Development plan. Write a one-year plan. Rubric: 3 = specific activities, sources, ethics, measurable goals; 2 = three; 1 = vague; 0 = none.
Synthesis
Capstone assessment shows that your claims follow from evidence, that you know what remains risky and that you have a plan to keep growing.
Continue studying — chapter 30 of 30
← Chapter 29 (draft) · Course chapter index · Worked results ↓ · Next course via chapter index →