Chapter 12 — Final review and transfer
Lesson map
Lesson 12 published. Map, example, EX056.
flowchart TD
N0["Learning outcomes"]
N1["12.1 Principles to keep"]
N2["12.2 Transfer to new technology"]
N3["12.3 Personal practice plan"]
N4["12.4 Mentoring"]
N5["12.5 Readiness check"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX056"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["12.1 Principles to keep"]
N2["12.2 Transfer to new technology"]
N3["12.3 Personal practice plan"]
N4["12.4 Mentoring"]
N5["12.5 Readiness check"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX056"]
[Learning outcomes]
|
v
[12.1 Principles to keep]
|
v
[12.2 Transfer to new technology]
|
v
[12.3 Personal practice plan]
|
v
[12.4 Mentoring]
|
v
[12.5 Readiness check]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX056]
Learning outcomes
You will (1) review the course's main principles; (2) transfer them to new AI technologies; (3) build a personal practice plan; (4) mentor others; and (5) assess your own readiness to lead.
12.1 Principles to keep
Decide among options with explicit uncertainty. Map each system to taxonomies, then to its own data, tools and affected people. Assume models can be steered; enforce consequences outside them. Evaluate adaptively, measure effects and report intervals. Integrate AI into existing processes. Assure claims with dated evidence and keep them current. Read research for its threat model.
Each principle appears in Corvane's work: the refund decision record, the copilot mapping, the policy service and the dated assurance case. Use them as a checklist for new work.
12.2 Transfer to new technology
New capabilities — computer-use agents, multimodal inputs, on-device models, model-to-model protocols — will keep arriving. Ask the same questions: what untrusted input reaches the model? What can it do? What enforces limits outside it? What evidence shows the limits hold? Multimodal inputs, for example, extend the injection channel to images and audio; the principle of separating untrusted content from authority still applies.
Amira's computer-use agent in the worked example below shows the transfer: portal pages are untrusted input and a proxy enforces listed operations.
12.3 Personal practice plan
Plan for the next year: one system you will assure end to end; one evaluation you will automate; one paper a month; one community contribution (a test suite, a talk, a framework comment); and ethics commitments — authorised testing, coordinated disclosure, honest reporting.
Make each item measurable, with a date. For example, one system assured end to end could be a high-tier system at your organisation with a dated assurance case and an evaluation that re-runs automatically. Review the plan with a peer every quarter.
12.4 Mentoring
Teach juniors to ask "what can it do?" before "what can it say?". Review their evaluations for adaptive cases and intervals. Share decision records so they see reasoning, not just outcomes.
"""Lab B: base rates and alarm precision (Bayes' rule).
A detector for malicious prompts with given true- and false-positive rates
is applied to traffic with different shares of malicious prompts."""
for tpr, fpr in ((0.95, 0.05), (0.95, 0.01), (0.80, 0.001)):
for prev in (0.10, 0.01, 0.001):
tp = tpr * prev; fp = fpr * (1 - prev)
prec = tp / (tp + fp)
per_10k = round(10000 * (tp + fp))
print(f"tpr={tpr:.2f} fpr={fpr:.3f} prevalence={prev:.3f} precision={prec:.3f} alerts_per_10k={per_10k}")
Expected output:
tpr=0.95 fpr=0.050 prevalence=0.100 precision=0.679 alerts_per_10k=1400
tpr=0.95 fpr=0.050 prevalence=0.010 precision=0.161 alerts_per_10k=590
tpr=0.95 fpr=0.050 prevalence=0.001 precision=0.019 alerts_per_10k=509
tpr=0.95 fpr=0.010 prevalence=0.100 precision=0.913 alerts_per_10k=1040
tpr=0.95 fpr=0.010 prevalence=0.010 precision=0.490 alerts_per_10k=194
tpr=0.95 fpr=0.010 prevalence=0.001 precision=0.087 alerts_per_10k=109
tpr=0.80 fpr=0.001 prevalence=0.100 precision=0.989 alerts_per_10k=809
tpr=0.80 fpr=0.001 prevalence=0.010 precision=0.890 alerts_per_10k=90
tpr=0.80 fpr=0.001 prevalence=0.001 precision=0.445 alerts_per_10k=18
At Corvane, Tomás's placement shows the point: he learned from Amira's decision records and reviews, then wrote his own harness and runbook. Ask juniors to explain their reasoning before you give yours, and review their work against the same principles you apply to your own. Let them make small mistakes in safe settings.
12.5 Readiness check
Can you write a decision record a board could act on? Can you design an evaluation that would change a decision? Can you explain why a 95%-accurate detector can produce mostly false alarms (the lab: at prevalence 0.1%, precision is 1.9%)? Can you lead a contested decision without hiding uncertainty?
Answer each with an example from your own work, such as a decision record or an evaluation, and ask a peer to judge whether it would convince a board. Where you cannot, plan how to build that skill this year.
12.6 Worked example (synthetic Corvane case)
A year on, Corvane plans a computer-use agent to operate a legacy carrier portal. Amira applies the same questions: the portal's pages are untrusted input; the agent's clicks are actions; enforcement must sit in a proxy that permits only listed operations with the user's credentials; evaluation needs adaptive pages with hidden instructions. Her decision record proposes a three-month pilot on read-only operations first.
The proposal follows the same reasoning as the refund and rebooking decisions. Evaluation and enforcement are planned before any write operation is allowed.
12.7 Errors and limits
- Treating each new technology as needing new principles.
- Plans without measurable commitments.
- Mentoring that transfers answers instead of reasoning.
Each error weakens transfer. New technologies need the same questions about untrusted input and enforcement; inventing new principles each time loses what was learned. A plan without measurable commitments is easily dropped; give each item a date and review the plan with a peer every quarter. Mentoring that hands over answers leaves juniors unable to reason about the next system. Check your practice plan for all three.
Exercises
- EX056 — Principles. State the five principles you would teach first, with one sentence each. Rubric: 3 = five apt; 2 = four; 1 = two; 0 = none.
- EX057 — Transfer. Apply the four questions to a voice assistant that can place orders. Rubric: 3 = all four answered concretely; 2 = three; 1 = one or two; 0 = none.
- EX058 — Base rate. Using the lab, explain the alert burden for a detector with 1% false positives at 1% prevalence. Rubric: 3 = precision 0.49, 194 alerts per 10k, implication; 2 = numbers only; 1 = one; 0 = wrong.
- EX059 — Practice plan. Write your one-year plan. Rubric: 3 = five measurable items including ethics; 2 = four; 1 = two; 0 = none.
- EX060 — Pilot. Justify read-only first for the computer-use agent. Rubric: 3 = reversibility, evidence gathering, bounded harm, triggers; 2 = three; 1 = one; 0 = none.
Synthesis
The principles of this course transfer to new AI technologies because they concern authority, evidence and accountability rather than any single model. Keep practising, mentoring and revising them.
Continue studying — chapter 12 of 12
← Chapter 11 (draft) · Course chapter index · Worked results ↓ · Next course via chapter index →