Chapter 11 — Supervised laboratory

Lesson map

Lesson 11 published. Map, example, EX051.

flowchart TD
  N0["Learning outcomes"]
  N1["11.1 Setup"]
  N2["11.2 Part A — assistant"]
  N3["11.3 Part B — scoring API"]
  N4["11.3b Adaptive round"]
  N5["11.4 Side effects"]
  N6["11.5 Report"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX051"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["11.1 Setup"]
  N2["11.2 Part A — assistant"]
  N3["11.3 Part B — scoring API"]
  N4["11.3b Adaptive round"]
  N5["11.4 Side effects"]
  N6["11.5 Report"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX051"]

[Learning outcomes]
    |
    v
[11.1 Setup]
    |
    v
[11.2 Part A — assistant]
    |
    v
[11.3 Part B — scoring API]
    |
    v
[11.3b Adaptive round]
    |
    v
[11.4 Side effects]
    |
    v
[11.5 Report]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX051]

Learning outcomes

You will (1) run a supervised end-to-end lab on an AI-service copy; (2) test, configure and re-test controls; (3) observe privacy and extraction risks in toy models; (4) measure side effects; and (5) report results with evidence.

11.1 Setup

The course environment provides an assistant copy with document upload, retrieval and two tools, plus a scoring API. Synthetic accounts and documents only. Your supervisor approves the plan before you start.

Keep a lab log from the start. Read the lab guide, check that your accounts and documents are synthetic, and confirm the tools in the copy cannot reach real systems. Run a short smoke test to check that the harness records inputs, outputs and tool calls before starting Part A.

11.2 Part A — assistant

Run the injection, disclosure and agency suites on the default configuration. Configure controls from Chapter 7. Re-run, including held-out cases. Measure normal-task completion before and after.

Use the controls you configured in Chapter 7 as a model. Record the results before and after, with intervals, and note any change in normal-task completion, since a control that blocks attacks but breaks normal use needs adjusting. Run the held-out cases only once, at the end. Note each setting changed. Run baseline tests on the assistant copy with no controls, then add controls one at a time and re-test.

11.3 Part B — scoring API

Run the privacy and extraction labs.

"""Lab M: membership inference against an overfit 1-nearest-neighbour 'model'
that returns a confidence score. Synthetic records only; teaching model."""
import random, math
rng = random.Random(5)
def rec(): 
    y = rng.random() < 0.5
    return [rng.gauss(0.6 if y else -0.6, 1.0) for _ in range(4)], int(y)
members = [rec() for _ in range(150)]; nonmembers = [rec() for _ in range(150)]
def confidence(x, y, k):
    # k-NN vote share for the true label, computed over the training set
    d = sorted((sum((a-b)**2 for a, b in zip(x, m)), my) for m, my in members)
    return sum(1 for _, my in d[:k] if my == y)/k
def auc(pos, neg):
    s = sum((p > n) + 0.5*(p == n) for p in pos for n in neg)
    return s/(len(pos)*len(neg))
for k in (1, 5, 25):
    pos = [confidence(x, y, k) for x, y in members]
    neg = [confidence(x, y, k) for x, y in nonmembers]
    best = max(sum(p >= t for p in pos)/len(pos) - sum(n >= t for n in neg)/len(neg)
               for t in sorted(set(pos + neg)))
    print(f"k={k:2d} attack_auc={auc(pos, neg):.3f} best_advantage={best:.3f}")

Expected output:

k= 1 attack_auc=0.600 best_advantage=0.200
k= 5 attack_auc=0.521 best_advantage=0.073
k=25 attack_auc=0.520 best_advantage=0.060

"""Lab X: model extraction of a toy linear scoring API by querying it.
The 'API' returns a real-valued score; d+1 well-chosen queries recover the
weights exactly, and rounding the output limits fidelity. Synthetic model."""
import random
rng = random.Random(9)
d = 5
secret_w = [round(rng.uniform(-2, 2), 3) for _ in range(d)]; secret_b = 0.42
def api(x, decimals=None):
    s = sum(a*b for a, b in zip(secret_w, x)) + secret_b
    return round(s, decimals) if decimals is not None else s
for dec in (None, 2, 0):
    b = api([0]*d, dec)
    w = [api([1 if j == i else 0 for j in range(d)], dec) - b for i in range(d)]
    err = max(abs(a-c) for a, c in zip(w, secret_w))
    test = [[rng.uniform(-1, 1) for _ in range(d)] for _ in range(500)]
    agree = sum((sum(a*c for a, c in zip(w, x))+b > 0) == (api(x) > 0) for x in test)/500
    print(f"output_rounding={str(dec):4s} queries={d+1} max_weight_error={err:.3f} label_agreement={agree:.3f}")

Expected output:

output_rounding=None queries=6 max_weight_error=0.000 label_agreement=1.000
output_rounding=2    queries=6 max_weight_error=0.004 label_agreement=0.998
output_rounding=0    queries=6 max_weight_error=0.534 label_agreement=0.882

Membership advantage is highest (0.20) when the toy model effectively memorises its training records (k = 1) and falls as it generalises; exact scores let six queries copy the toy model, while whole-number rounding raises error to 0.534. Relevant settings: coarse outputs, rate limits, monitoring.

Record outputs for each setting. Membership tests show how much a model reveals about its training records, and extraction tests show how many queries an attacker needs to copy it. Compare exact and coarse outputs, and note how rate limits would change the attacker's cost.

11.3b Adaptive round

After the planned suites, spend a supervised hour trying new ways around your controls, knowing how they work: splitting a request across turns, hiding text in images, or using another language. Record every attempt, successful or not. Adaptive attempts often find gaps fixed suites miss, and they show the supervisor how you reason as an attacker within safe limits.

Note each attempt's result in the lab log. After your controls are in place, try new attack variations that you did not use while tuning, as an attacker would. Record whether any succeed and what you changed in response, then re-run the held-out cases.

11.4 Side effects

Controls can block normal use. Record false refusals and completion rates; adjust where the cost is too high, and note the trade-off.

In Sofia's lab, completion fell from 47 to 46 of 50 after controls, a small cost she recorded. If a control blocks normal use more than that, adjust it and re-test, and explain the trade-off in the report so the supervisor can judge it. Also check whether any control slows replies. Run the normal-task suite after each control is added, so you can see which one causes any drop in completion.

11.5 Report

Table per category: before, after, held-out, intervals; completion; remaining failures; recommendations; what was not tested.

Sofia's report followed this table and named image-only uploads as untested. Add intervals for held-out results, as her supervisor asked, and keep the report short enough to read in ten minutes. Keep it to one page plus the table. State each rate as counts with an interval, such as 0 of 25 held-out cases. List remaining failures with case IDs, recommendations with the risk each addresses, and untested areas, such as image-only uploads, so the next tester knows where to start. Your supervisor signs off the report.

11.6 Worked example (synthetic Meridian case)

Sofia's lab: injection-caused actions 17/50 → 0/50 (held-out 0/25); disclosure 7/20 → 0/20; agency 5/15 → 0/15; completion 47/50 → 46/50. Part B: she recommends FraudFlag's partner API return only "standard / review" and adds per-partner rate limits. Not tested: image-only uploads with text in pictures. Her supervisor signs off the report with one request: add intervals for the held-out results.

The report states what was not tested, which is as useful to the next tester as the results. Sofia's results show the controls worked: injection fell from 17 of 50 to 0 of 50, with 0 of 25 held-out cases succeeding.

11.7 Errors and limits

• Skipping the baseline.
• Ignoring side effects.
• Reports without "not tested".

Each error weakens the lab. Without a baseline, there is no way to show a control helped. Ignoring side effects hides the cost to normal users. A report without a "not tested" list implies coverage that does not exist; Sofia named image-only uploads for this reason. Check your lab report for all three before submitting it. Fix any before your supervisor reviews the report. Sofia's report avoided all three, and her supervisor asked only for intervals on held-out results.

Exercises

• EX051 — Plan. Write the lab plan for supervisor approval. Rubric: 3 = scope, suites, controls, measures, safety; 2 = four; 1 = two; 0 = none.
• EX052 — Results. Build the results table for Part A. Rubric: 3 = categories, before, after, held-out, completion; 2 = four; 1 = two; 0 = none.
• EX053 — Interval. Give the upper bound for 0/25 held-out. Rubric: 3 = ≈12% with meaning; 2 = number; 1 = vague; 0 = wrong.
• EX054 — Part B. Explain two Part B numbers. Rubric: 3 = two correct with relevance; 2 = one; 1 = vague; 0 = none.
• EX055 — Not tested. Why include "not tested" in the report? Rubric: 3 = honest scope, prevents overclaiming, plans next work; 2 = two; 1 = one; 0 = none.

Synthesis

The supervised lab rehearses the whole practice cycle — baseline, configure, re-test, measure side effects, report honestly — in a safe setting.

Continue studying — chapter 11 of 12

← Chapter 10 (draft) · Course chapter index · Chapter 12 (draft) →