Chapter 18 — Supervised technical project
Lesson map
Lesson 18 published. Map, example, EX069.
flowchart TD
N0["Learning outcomes"]
N1["18.1 Project brief"]
N2["18.2 Planning"]
N3["18.3 Verification and evidence"]
N4["18.4 Handover"]
N5["18.5 Reflection"]
N6["18.5b Assessment"]
N7["18.7b Presenting the project"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX069"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["18.1 Project brief"]
N2["18.2 Planning"]
N3["18.3 Verification and evidence"]
N4["18.4 Handover"]
N5["18.5 Reflection"]
N6["18.5b Assessment"]
N7["18.7b Presenting the project"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX069"]
[Learning outcomes]
|
v
[18.1 Project brief]
|
v
[18.2 Planning]
|
v
[18.3 Verification and evidence]
|
v
[18.4 Handover]
|
v
[18.5 Reflection]
|
v
[18.5b Assessment]
|
v
[18.7b Presenting the project]
|
v
[Worked example] --> [Exercise EX069]
Learning outcomes
You will (1) complete a supervised AI-security project on a real or realistic service; (2) plan, configure, verify and document controls; (3) hand over to operations; (4) reflect on your practice.
18.1 Project brief
With a supervisor and written authorisation, choose one AI service (or a staging copy). Deliver: system model and risks; configured controls; test suite and results; monitoring checks; runbooks; documentation; handover meeting. Agree the scope at the start and record any changes.
Ana's authorisation for the SpamShield project named the partner API's staging copy, the synthetic partner keys she could use, the activities (extraction-pattern queries, simulated feedback poisoning, load tests) and her supervisor. Mid-project the scope gained a load test of the partner rate limits; she recorded the change and its approval before running it, because a load test against the wrong environment would affect real partners.
18.2 Planning
Weeks 1–2: model the system, list risks, agree scope. Weeks 3–5: configure and test controls. Week 6: monitoring and runbooks. Week 7: documentation and handover. Meet the supervisor weekly with the latest test report.
For SpamShield, Ana's weeks one and two produced the system model and the four risks; weeks three to five covered category-only output, per-partner limits, feedback weight caps and their tests; week six the volume-outlier alert and the runbook; week seven the documentation and the handover drill. Each weekly meeting began with the latest harness report, so her supervisor could see progress in test results rather than in descriptions.
18.3 Verification and evidence
Map each risk to a control and a test. Keep results with versions and dates. Report remaining failures honestly.
Ana's mapping table for SpamShield had four rows. Extraction maps to category-only output and rate limits, tested by 50 extraction-pattern sequences. Evasion maps to the scam-pattern test set, tested on each model update. Poisoning maps to feedback weight caps and review, tested by a simulated bulk-feedback attack. Consumption maps to per-partner limits, tested by a load run. Each row carried versions, dates and results, including the scam patterns the model still misclassified.
18.4 Handover
Walk the operations team through controls, alerts and runbooks. Let them run one containment step themselves. Leave the test suite running in the pipeline.
Ana's handover for SpamShield covered the partner-volume alert, the feedback review queue and the rate-limit runbook. The drill had an operator rate-limit a synthetic test partner from the runbook alone, which took four minutes. The test suite stayed in the pipeline, so the next model update would be checked whether or not Ana was still on the team.
Operators also found the feedback review queue, holding one simulated bulk batch, without help.
18.5 Reflection
What worked, what you would do differently, what you still need to learn. Keep ethics in view: authorised work only, protect customer data, report problems honestly.
"""Lab B: base rates and alarm precision (Bayes' rule).
A detector for malicious prompts with given true- and false-positive rates
is applied to traffic with different shares of malicious prompts."""
for tpr, fpr in ((0.95, 0.05), (0.95, 0.01), (0.80, 0.001)):
for prev in (0.10, 0.01, 0.001):
tp = tpr * prev; fp = fpr * (1 - prev)
prec = tp / (tp + fp)
per_10k = round(10000 * (tp + fp))
print(f"tpr={tpr:.2f} fpr={fpr:.3f} prevalence={prev:.3f} precision={prec:.3f} alerts_per_10k={per_10k}")
Expected output:
tpr=0.95 fpr=0.050 prevalence=0.100 precision=0.679 alerts_per_10k=1400
tpr=0.95 fpr=0.050 prevalence=0.010 precision=0.161 alerts_per_10k=590
tpr=0.95 fpr=0.050 prevalence=0.001 precision=0.019 alerts_per_10k=509
tpr=0.95 fpr=0.010 prevalence=0.100 precision=0.913 alerts_per_10k=1040
tpr=0.95 fpr=0.010 prevalence=0.010 precision=0.490 alerts_per_10k=194
tpr=0.95 fpr=0.010 prevalence=0.001 precision=0.087 alerts_per_10k=109
tpr=0.80 fpr=0.001 prevalence=0.100 precision=0.989 alerts_per_10k=809
tpr=0.80 fpr=0.001 prevalence=0.010 precision=0.890 alerts_per_10k=90
tpr=0.80 fpr=0.001 prevalence=0.001 precision=0.445 alerts_per_10k=18
Ana's reflection named specific actions. What worked: running baselines first, which caught a broken grader in week three. What she would change: involving the data team earlier, since the feedback weight cap needed their training pipeline. What to learn next: how evasion testing works on real text classifiers rather than the toy model. On ethics, she noted that every test ran on synthetic partners, which was in the authorisation, and that one remaining evasion gap was reported, not hidden.
18.5b Assessment
Supervisors assess traceability from risk to evidence, correct and safe configuration, honest reporting, operational readiness and clear documentation.
For SpamShield, traceability meant each of the four risks could be followed to a control and a test result; operational readiness was shown by the four-minute drill. Honest reporting meant the scam patterns SpamShield still misclassified were in the report's first table, not in an appendix. A project with impressive controls but no evidence linking them to risks would not pass.
Documentation was judged by the drill: the operator rate-limited the test partner without asking Ana anything.
18.6 Worked example (synthetic Kestrel case)
Ana's project secures SpamShield's partner API. Risks: extraction, evasion, poisoning via feedback, consumption. Controls: category-only output, per-partner rate limits, feedback weight caps with review, alerts on volume outliers. Tests: 50 extraction-pattern sequences blocked by limits; feedback-poisoning simulation rejected at review. She chooses an alert threshold using base rates: at 1% prevalence and 1% false positives, about half of alerts are real, which the team can handle. Handover drill: an operator rate-limits a test partner in four minutes.
The threshold choice came from the base-rate lab: 0.95 detection with 1% false positives at 1% prevalence gives precision 0.490.
18.7 Errors and limits
- Scope changing without record.
- Handover by email only.
- Reflection without specific actions.
Each error has a SpamShield form. If the load test had been added without recording the scope change, the authorisation would no longer have covered it. A handover by email would have left the operator who ran the four-minute drill reading a document for the first time during a real partner surge. A reflection such as "testing went well" gives the next project nothing to change; Ana's said to involve the data team in week one.
18.7b Presenting the project
The supervised project ends with a ten-minute presentation to the team lead: what was built, how it was verified, what is still open and who now owns it. The lead signs the handover only when the new owner confirms they can run it alone.
Ana's presentation covered SpamShield's four risks and their evidence, the evasion gap still open, and the operations team as new owner. The lead signed after the operator who had done the drill confirmed the runbook was enough.
The open evasion gap went to the data team with the scam-pattern test set attached.
Exercises
- EX069 — Plan. Write a seven-week plan for a HelpBot project. Rubric: 3 = phases, deliverables, meetings; 2 = two; 1 = vague; 0 = none.
- EX070 — Mapping. Map SpamShield's four risks to controls and tests. Rubric: 3 = four complete rows; 2 = three; 1 = one or two; 0 = none.
- EX071 — Threshold. Explain Ana's alert-threshold reasoning using the lab. Rubric: 3 = precision 0.49 at 1%/1%, workload acceptable; 2 = numbers; 1 = vague; 0 = wrong.
- EX072 — Reflection. Write three reflection points with actions. Rubric: 3 = three with actions; 2 = two; 1 = one; 0 = none.
Synthesis
The supervised project brings the course together: authorised scope, configured and verified controls, monitoring, documentation and a real handover.
Continue studying — chapter 18 of 18
← Chapter 17 (draft) · Course chapter index · Worked results ↓ · Next course via chapter index →