Chapter 7 — AI and machine-learning security: data and telemetry
Lesson map
Lesson 07 published. Map, example, EX025.
flowchart TD
N0["Learning outcomes"]
N1["7.1 Data an AI service uses"]
N2["7.2 Controlling changes"]
N3["7.3 Telemetry"]
N4["7.4 Personal data in telemetry"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX025"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["7.1 Data an AI service uses"]
N2["7.2 Controlling changes"]
N3["7.3 Telemetry"]
N4["7.4 Personal data in telemetry"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX025"]
[Learning outcomes]
|
v
[7.1 Data an AI service uses]
|
v
[7.2 Controlling changes]
|
v
[7.3 Telemetry]
|
v
[7.4 Personal data in telemetry]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX025]
Learning outcomes
You will (1) manage the data an AI service uses — training sets, indexes, logs; (2) control who can change it; (3) collect telemetry needed for security; (4) protect personal data in telemetry.
7.1 Data an AI service uses
SpamShield learns from labelled messages; HelpBot retrieves from an index of help articles; both produce logs. Each data set has an owner, a sensitivity class and a list of people who can change it. Changes to training data and indexes change the system's behaviour as surely as code changes do.
At Kestrel the owners are concrete. The help-article index belongs to the content team, and only content owners may publish; the factory-reset incident started when a contractor account published without review. SpamShield's training labels belong to the data team and include customer feedback, which is how 900 bulk "not spam" reports reached a training run. LineWatch's telemetry belongs to network operations. Ana's inventory listed each set with its owner, its sensitivity and the accounts able to change it.
7.2 Controlling changes
Restrict who can add or edit indexed documents and training labels. Record every change with who, when and what. Review new sources before indexing. Poisoning attacks — corrupting labels or planting backdoor triggers (Gu et al., arXiv:1708.06733) — rely on loose change control.
"""Lab P: label-flip poisoning and a backdoor trigger on a toy nearest-centroid
classifier (synthetic data, stdlib only). Teaching model only."""
import random
rng = random.Random(11)
def point(label, trigger=False):
cx = 1.0 if label else -1.0
x = [rng.gauss(cx, 0.8), rng.gauss(cx, 0.8), 0.0]
if trigger: x[2] = 5.0 # third feature is the 'trigger'
return x, label
clean = [point(i % 2) for i in range(400)]
test = [point(i % 2) for i in range(400)]
def fit(data):
c = {}
for lab in (0, 1):
pts = [x for x, y in data if y == lab]
c[lab] = [sum(p[i] for p in pts)/len(pts) for i in range(3)]
return c
def pred(c, x):
d = {k: sum((x[i]-v[i])**2 for i in range(3)) for k, v in c.items()}
return min(d, key=d.get)
def accuracy(c, data): return sum(pred(c, x) == y for x, y in data)/len(data)
print("label-flip poisoning (flip a fraction of class-1 labels to 0):")
for frac in (0.0, 0.1, 0.2, 0.3, 0.4):
data = [(x, y) for x, y in clean]; ones = [i for i, (x, y) in enumerate(data) if y == 1]
for i in ones[:int(frac*len(ones))]: data[i] = (data[i][0], 0)
print(f" flipped={frac:.1f} clean_test_accuracy={accuracy(fit(data), test):.3f}")
print("backdoor: add n triggered class-1 points labelled 0")
trig_test = [point(1, trigger=True) for _ in range(200)]
for n in (0, 10, 40, 80):
data = clean + [(point(1, trigger=True)[0], 0) for _ in range(n)]
c = fit(data)
asr = sum(pred(c, x) == 0 for x, _ in trig_test)/len(trig_test)
print(f" poison_points={n:3d} clean_accuracy={accuracy(c, test):.3f} attack_success={asr:.3f}")
Expected output:
label-flip poisoning (flip a fraction of class-1 labels to 0):
flipped=0.0 clean_test_accuracy=0.965
flipped=0.1 clean_test_accuracy=0.965
flipped=0.2 clean_test_accuracy=0.953
flipped=0.3 clean_test_accuracy=0.950
flipped=0.4 clean_test_accuracy=0.945
backdoor: add n triggered class-1 points labelled 0
poison_points= 0 clean_accuracy=0.965 attack_success=0.075
poison_points= 10 clean_accuracy=0.963 attack_success=0.145
poison_points= 40 clean_accuracy=0.963 attack_success=0.685
poison_points= 80 clean_accuracy=0.968 attack_success=0.980
After the SpamShield incident, Kestrel's rule for training data became: feedback from one account counts for at most a fixed weight, bulk feedback above a threshold is held for review, and every training run records which feedback batches it used. For the article index, every publish records the author account and the reviewer. These records let Ana find the 900 reports and the contractor article quickly; without them, both investigations would have started from the symptoms alone.
7.3 Telemetry
Collect: request counts and tokens per account; retrieved document IDs per answer; tool calls with parameters and results; policy decisions; model and prompt versions; errors. For classifiers: score distributions and label feedback. Use a shared request ID so one conversation can be followed end to end.
Each field answered a question in a Kestrel incident. Retrieved document IDs per answer pointed to the factory-reset article within an hour. Tool calls with parameters revealed the three accounts with more than 50 restarts. Tokens per account showed the 40,000-message weekend. Model and prompt versions ruled out a model change in the factory-reset case. For SpamShield, label feedback by account showed the 900 reports. The shared request ID is what lets one technician join these records for a single conversation.
7.4 Personal data in telemetry
Logs often contain names, addresses and account numbers. Redact or pseudonymise where possible, restrict access, and set retention periods. Telemetry that leaks customer data creates the problem it was meant to detect.
For HelpBot, Ana's proposal kept account IDs but replaced names, street addresses and phone numbers in conversation text with placeholders before logs left the orchestrator, limited raw conversation access to the on-call lead, and set 30 days' retention for conversation text and longer for tool-call records. A customer who types a home address to report a fault should not find it in a dashboard that dozens of staff can read. The restart counter and canary searches both worked on the redacted logs.
7.5 Worked example (synthetic Kestrel case)
SpamShield's accuracy is steady, but customers complain about one type of scam message getting through. Ana checks the label feedback: an account had submitted 900 "not spam" reports for messages containing a particular short link. The lab shows how a backdoor can leave clean accuracy unchanged while making triggered inputs pass. The team limits feedback weight per account, reviews bulk feedback before training, and adds a test set of known scam patterns.
The lab explains why accuracy hid the problem. With 80 triggered poison points, clean accuracy stayed at 0.968 while the trigger succeeded 98% of the time; with 10 points it rose only to 14.5%, against a 7.5% baseline, and with 40 points it jumped to 68.5%. A volume of 900 reports from one account is far above that range. Overall accuracy is therefore not a safety check: only a test set containing the scam pattern itself would have shown the failure.
7.6 Errors and limits
- Indexes anyone can edit.
- Telemetry without request IDs.
- Unredacted personal data in logs.
Each of these hit Kestrel. An index anyone could edit let a contractor publish the factory-reset article and let another edit tell HelpBot to restart routers five times. Without request IDs, linking a bad answer to the article it retrieved would have meant matching timestamps by hand across three logs. Unredacted personal data in logs turns every troubleshooting session into a privacy exposure; the redacted copies were enough for every check described in this chapter.
Exercises
- EX025 — Change control. Write a change-control rule for HelpBot's article index. Rubric: 3 = who can edit, review, record, rollback; 2 = three; 1 = one or two; 0 = none.
- EX026 — Poison lab. Interpret the backdoor rows of the lab. Rubric: 3 = attack success rises, clean accuracy flat, implication; 2 = two; 1 = one; 0 = wrong.
- EX027 — Telemetry list. List six telemetry fields and why each matters. Rubric: 3 = six with reasons; 2 = four; 1 = two; 0 = none.
- EX028 — Redaction. Propose redaction and retention for HelpBot logs. Rubric: 3 = fields masked, access, retention period; 2 = two; 1 = one; 0 = none.
Synthesis
Data and telemetry shape and reveal AI behaviour: control who changes data, collect traceable telemetry and protect personal data within it.
Continue studying — chapter 07 of 18
← Chapter 06 (draft) · Course chapter index · Chapter 08 (draft) →