Chapter 14 — Privacy foundations
Lesson map
Lesson 14 published. Map, example, EX040.
flowchart TD
N0["Learning outcomes"]
N1["14.1 Principles"]
N2["14.2 Privacy attacks on ML"]
N3["14.3 Privacy-enhancing techniques"]
N4["14.4 Privacy risk assessment"]
N5["14.4b Prompts and logs"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX040"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["14.1 Principles"]
N2["14.2 Privacy attacks on ML"]
N3["14.3 Privacy-enhancing techniques"]
N4["14.4 Privacy risk assessment"]
N5["14.4b Prompts and logs"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX040"]
[Learning outcomes]
|
v
[14.1 Principles]
|
v
[14.2 Privacy attacks on ML]
|
v
[14.3 Privacy-enhancing techniques]
|
v
[14.4 Privacy risk assessment]
|
v
[14.4b Prompts and logs]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX040]
Learning outcomes
You will (1) explain data-protection principles and personal data; (2) describe privacy attacks on machine learning (membership inference, data extraction, attribute inference); (3) apply privacy-enhancing techniques; and (4) assess privacy risks of an AI system.
14.1 Principles
Data-protection laws such as the EU GDPR require a lawful basis, purpose limitation, data minimisation, accuracy, storage limitation, security and accountability. Health data is a special category with stricter rules.
For Ask Lumen, purpose limitation means conversation logs kept for quality review are not reused to train marketing models; minimisation means the prompt includes only what is needed to answer; storage limitation means logs are deleted after their retention period. Because health data is a special category, LCH's lawful basis and safeguards are documented in its impact assessment before any new use.
14.2 Privacy attacks on ML
Membership inference decides whether a record was in training data (Shokri et al., arXiv:1610.05820). Training-data extraction recovers memorised text from generative models (Carlini et al., arXiv:2012.07805). Attribute inference guesses sensitive attributes from model outputs. Overfitting increases these risks; the lab shows membership leaking from a memorising model and fading as it generalises.
"""Lab M: membership inference against an overfit 1-nearest-neighbour 'model'
that returns a confidence score. Synthetic records only; teaching model."""
import random, math
rng = random.Random(5)
def rec():
y = rng.random() < 0.5
return [rng.gauss(0.6 if y else -0.6, 1.0) for _ in range(4)], int(y)
members = [rec() for _ in range(150)]; nonmembers = [rec() for _ in range(150)]
def confidence(x, y, k):
# k-NN vote share for the true label, computed over the training set
d = sorted((sum((a-b)**2 for a, b in zip(x, m)), my) for m, my in members)
return sum(1 for _, my in d[:k] if my == y)/k
def auc(pos, neg):
s = sum((p > n) + 0.5*(p == n) for p in pos for n in neg)
return s/(len(pos)*len(neg))
for k in (1, 5, 25):
pos = [confidence(x, y, k) for x, y in members]
neg = [confidence(x, y, k) for x, y in nonmembers]
best = max(sum(p >= t for p in pos)/len(pos) - sum(n >= t for n in neg)/len(neg)
for t in sorted(set(pos + neg)))
print(f"k={k:2d} attack_auc={auc(pos, neg):.3f} best_advantage={best:.3f}")
Expected output:
k= 1 attack_auc=0.600 best_advantage=0.200
k= 5 attack_auc=0.521 best_advantage=0.073
k=25 attack_auc=0.520 best_advantage=0.060
Lab M makes the effect measurable: with k of 1 the attack reaches an AUC of 0.600, while with k of 25 it falls to 0.520, close to chance. Averaging over more neighbours, as a model that generalises does, reduces what the output reveals about individual patients.
14.3 Privacy-enhancing techniques
Minimise and pseudonymise data; deduplicate training records; limit output detail (labels rather than confidences); apply differential privacy during training (Abadi et al., arXiv:1607.00133); control who can query models; and never put personal data into prompts unnecessarily.
Each technique targets a part of the risk. Deduplication removes repeated records that models tend to memorise. Returning labels rather than confidences removes the signal Lab M's attack uses. Differential privacy bounds how much any one record can affect the trained model, at some cost in accuracy. Query controls limit who can probe the model at all. LCH's NoShow change used four of these together.
14.4 Privacy risk assessment
Ask: what personal data enters training, retrieval, prompts and logs? Who can query the model, and what do outputs reveal? What happens if logs leak? A data-protection impact assessment documents the answers and controls.
For NoShow, the answers showed that personal data entered training, that any staff member could query any patient ID, and that outputs gave precise probabilities. Each finding led to a control: restricted queries, category outputs and minimised features. The assessment is updated when the system changes, not filed once and forgotten.
14.4b Prompts and logs
If staff paste patient details into an AI assistant, those details may be stored by the provider or appear in logs. Configure retention and training opt-outs where the provider offers them, redact identifiers before sending when possible, and train staff on what may be shared.
Redaction can be automated for structured identifiers such as patient numbers, but free text needs care: a patient may type a diagnosis or a relative's name anywhere in a message.
14.5 Worked example (synthetic LCH case)
NoShow's API returned a precise probability for any patient ID. Rina showed on synthetic data that confidence differed between patients in the training set and new patients, enabling membership inference ("was this person treated at LCH last year?"). LCH changed the API to return only a reminder category to authorised staff and retrained with deduplicated, minimised features.
Returning a category rather than a probability removed most of the signal, and limiting access to authorised staff reduced who could ask the question at all.
14.6 Errors and limits
- Treating models as if they contained no personal data.
- Detailed confidence outputs to broad audiences.
- Logs full of personal data with no retention limit.
Models trained on patient records carry information about those records, as membership inference shows. Detailed confidence scores to broad audiences give attackers the signal they need. Logs full of personal data with no retention limit become a store of sensitive information that grows every day. LCH's changes to NoShow and its log retention rules address all three, and the impact assessment records why.
Exercises
- EX040 — Lab reading. What AUC and advantage does the lab report at k = 1, and what does it mean? Rubric: 3 = 0.600 and 0.200; memorisation leaks membership; 2 = numbers; 1 = vague; 0 = wrong.
- EX041 — Sensitive membership. Why is membership in NoShow's training data sensitive? Rubric: 3 = reveals that a person was a patient (health information); 2 = partial; 1 = vague; 0 = wrong.
- EX042 — Techniques. List four privacy-enhancing techniques for NoShow. Rubric: 3 = four relevant; 2 = three; 1 = one or two; 0 = none.
Synthesis
Models can reveal their training data. Data-protection principles, privacy-aware outputs and techniques such as minimisation and differential privacy reduce the risk.
Continue studying — chapter 14 of 40
← Chapter 13 (draft) · Course chapter index · Chapter 15 (draft) →