Chapter 12 — Contained laboratory I
Lesson map
Lesson 12 published. Map, example, EX045.
flowchart TD
N0["Learning outcomes"]
N1["12.1 Lab rules"]
N2["12.2 Membership inference"]
N3["12.3 Model extraction"]
N4["12.4 Relating to configuration"]
N5["12.5 Lab record"]
N6["12.5b Sharing results"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX045"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["12.1 Lab rules"]
N2["12.2 Membership inference"]
N3["12.3 Model extraction"]
N4["12.4 Relating to configuration"]
N5["12.5 Lab record"]
N6["12.5b Sharing results"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX045"]
[Learning outcomes]
|
v
[12.1 Lab rules]
|
v
[12.2 Membership inference]
|
v
[12.3 Model extraction]
|
v
[12.4 Relating to configuration]
|
v
[12.5 Lab record]
|
v
[12.5b Sharing results]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX045]
Learning outcomes
You will (1) run contained labs safely; (2) observe evasion, membership inference and extraction in toy models; (3) relate each to service configuration; (4) write a lab record.
12.1 Lab rules
Run labs only on your own machine or the course environment, offline, with synthetic data. They are toy models to make ideas measurable, not tools for real systems.
For this course that means the lab files run with python3 and the standard library on the course machine, with no network calls and no real customer data. The extraction lab queries a scoring function defined in the same file, not SpamShield. Applying these techniques to Kestrel's partner API or to any real service would need separate written authorisation, and the course does not provide that. Results describe toy models built to make a mechanism measurable.
12.2 Membership inference
"""Lab M: membership inference against an overfit 1-nearest-neighbour 'model'
that returns a confidence score. Synthetic records only; teaching model."""
import random, math
rng = random.Random(5)
def rec():
y = rng.random() < 0.5
return [rng.gauss(0.6 if y else -0.6, 1.0) for _ in range(4)], int(y)
members = [rec() for _ in range(150)]; nonmembers = [rec() for _ in range(150)]
def confidence(x, y, k):
# k-NN vote share for the true label, computed over the training set
d = sorted((sum((a-b)**2 for a, b in zip(x, m)), my) for m, my in members)
return sum(1 for _, my in d[:k] if my == y)/k
def auc(pos, neg):
s = sum((p > n) + 0.5*(p == n) for p in pos for n in neg)
return s/(len(pos)*len(neg))
for k in (1, 5, 25):
pos = [confidence(x, y, k) for x, y in members]
neg = [confidence(x, y, k) for x, y in nonmembers]
best = max(sum(p >= t for p in pos)/len(pos) - sum(n >= t for n in neg)/len(neg)
for t in sorted(set(pos + neg)))
print(f"k={k:2d} attack_auc={auc(pos, neg):.3f} best_advantage={best:.3f}")
Expected output:
k= 1 attack_auc=0.600 best_advantage=0.200
k= 5 attack_auc=0.521 best_advantage=0.073
k=25 attack_auc=0.520 best_advantage=0.060
Membership inference asks whether a record was in the training data (Shokri et al., arXiv:1610.05820). In the lab, the attack's advantage is 0.20 when the score comes from the single nearest training record (k = 1) and falls to about 0.06 when it averages 25 neighbours. Service relevance: limit output detail (labels rather than exact scores) and keep training data minimal.
The k values show the mechanism. With k = 1 the score depends on a single training record, so a member's own record always votes for its label and the attacker gains an advantage of 0.20 over guessing. Averaging over 5 or 25 neighbours blurs that signal to 0.073 and 0.060. Exact scores give the attacker more to work with, which is why the SpamShield recommendation below returns categories instead.
12.3 Model extraction
"""Lab X: model extraction of a toy linear scoring API by querying it.
The 'API' returns a real-valued score; d+1 well-chosen queries recover the
weights exactly, and rounding the output limits fidelity. Synthetic model."""
import random
rng = random.Random(9)
d = 5
secret_w = [round(rng.uniform(-2, 2), 3) for _ in range(d)]; secret_b = 0.42
def api(x, decimals=None):
s = sum(a*b for a, b in zip(secret_w, x)) + secret_b
return round(s, decimals) if decimals is not None else s
for dec in (None, 2, 0):
b = api([0]*d, dec)
w = [api([1 if j == i else 0 for j in range(d)], dec) - b for i in range(d)]
err = max(abs(a-c) for a, c in zip(w, secret_w))
test = [[rng.uniform(-1, 1) for _ in range(d)] for _ in range(500)]
agree = sum((sum(a*c for a, c in zip(w, x))+b > 0) == (api(x) > 0) for x in test)/500
print(f"output_rounding={str(dec):4s} queries={d+1} max_weight_error={err:.3f} label_agreement={agree:.3f}")
Expected output:
output_rounding=None queries=6 max_weight_error=0.000 label_agreement=1.000
output_rounding=2 queries=6 max_weight_error=0.004 label_agreement=0.998
output_rounding=0 queries=6 max_weight_error=0.534 label_agreement=0.882
With exact scores, six queries recover the toy model exactly; rounding to two decimals keeps the error small (0.004); rounding to whole numbers raises it to 0.534. Service relevance: return coarse outputs, rate-limit and monitor scoring APIs (Tramèr et al., arXiv:1609.02943).
The rounding rows are the useful lesson. With exact scores, the six queries recover every weight with zero error and the copied model agrees with the original on all 500 test inputs. Rounding to two decimals changes almost nothing: error 0.004, agreement 0.998. Only rounding to whole numbers removes enough information to push the error to 0.534 and agreement down to 0.882. Mild rounding is therefore not a defence; returning only a category, as Ana recommended for SpamShield, removes much more.
12.4 Relating to configuration
Each lab suggests settings: output granularity, rate limits, monitoring of query patterns and access control on scoring endpoints.
For SpamShield's partner API, the labs translate into four settings. Return "spam / not spam / review" instead of probabilities, which removes both the membership signal and most of what extraction needs. Set per-partner rate limits, so recovering a model takes far longer than six queries. Alert on partners whose query volume is a far outlier, as in the robust z-score lab. Require a key per partner, so one misbehaving partner can be limited without affecting the others.
12.5 Lab record
Record: lab name, what was varied, results, what it shows, what it does not show, and the configuration it suggests.
Ana's record for the extraction lab: name, Lab X; varied, output rounding (none, two decimals, whole numbers); results, error 0.000, 0.004 and 0.534 with agreement 1.000, 0.998 and 0.882; shows, exact scores let a linear model be copied with d + 1 queries; does not show, how many queries a real non-linear model like SpamShield would need; suggests, category-only output, per-partner limits and volume alerts.
The "does not show" field is the one most often left blank, and it is the one that stops a toy result from being quoted as a fact about SpamShield.
12.5b Sharing results
Share lab records with your team in a short note: one table, three sentences on meaning and limits, and the recommended setting. Ask a colleague to rerun the lab to check that the outputs match. Labs whose results cannot be reproduced should not drive configuration changes.
Ana's extraction note was rerun by a colleague on a different laptop, and the three output lines matched exactly, which is what the fixed random seed in each lab guarantees.
When two runs disagree, the usual causes are a different Python version or an edited line, and both are worth finding before anyone changes a setting.
12.6 Worked example (synthetic Kestrel case)
SpamShield exposes an internal API returning exact spam probabilities to partner apps. After the extraction lab, Ana recommends returning only "spam / not spam / review", adding per-partner rate limits and alerting on partners whose query volume is far above normal.
The extraction lab drove the first change and the membership lab the second: exact probabilities are what both attacks feed on. The rate limits use the same token-bucket approach as HelpBot's gateway, set per partner key so one partner's surge cannot affect others. Ana did not claim SpamShield could be copied in six queries; she claimed only that exact scores make copying easier and that partners do not need them.
12.7 Errors and limits
- Running labs against real services.
- Reading toy numbers as real-world rates.
Running Lab X's query pattern against SpamShield's partner API would be unauthorised testing of a production service, whatever the intent. Reading toy numbers as real rates is the subtler error: the lab's 0.20 advantage comes from a 1-nearest-neighbour model on 150 synthetic records and says nothing about how much SpamShield leaks. The labs justify the direction of a setting, coarser outputs and tighter limits, not a claim about real-world risk.
Ana's recommendation note therefore said "exact scores make copying easier", never "SpamShield can be copied".
Exercises
- EX045 — MIA. Explain what "advantage 0.20" means. Rubric: 3 = attacker beats guessing by 20 points; 2 = partial; 1 = vague; 0 = wrong.
- EX046 — Extraction. Why does rounding to whole numbers increase error so much? Rubric: 3 = less information per query; 2 = partial; 1 = vague; 0 = wrong.
- EX047 — Configuration. Give three SpamShield API settings suggested by the labs. Rubric: 3 = coarse output, rate limit, monitoring; 2 = two; 1 = one; 0 = none.
- EX048 — Lab record. Write a lab record for the extraction lab. Rubric: 3 = all six fields; 2 = four; 1 = two; 0 = none.
Synthesis
Contained labs make privacy and extraction attacks visible and point directly to output, rate and monitoring settings.
Continue studying — chapter 12 of 18
← Chapter 11 (draft) · Course chapter index · Chapter 13 (draft) →