◇ OpenCyber Academy
Engineering · Chapter 13 · English

Chapter 13 — Data systems

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 13 — Data systems

Lesson map

Lesson 13 published. Map, example, EX037.

flowchart TD
  N0["Learning outcomes"]
  N1["13.1 Schemas, lineage and versions"]
  N2["13.2 Integrity of training data"]
  N3["13.3 Retrieval stores"]
  N4["13.4 Data protection"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX037"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["13.1 Schemas, lineage and versions"]
  N2["13.2 Integrity of training data"]
  N3["13.3 Retrieval stores"]
  N4["13.4 Data protection"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX037"]

[Learning outcomes]
    |
    v
[13.1 Schemas, lineage and versions]
    |
    v
[13.2 Integrity of training data]
    |
    v
[13.3 Retrieval stores]
    |
    v
[13.4 Data protection]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX037]

Learning outcomes

You will (1) model datasets with schemas, lineage and versions; (2) protect training and retrieval data against tampering and leakage; (3) design retrieval-augmented generation (RAG) stores with access control; and (4) apply data-protection principles to AI data.

13.1 Schemas, lineage and versions

Every dataset should have a schema, a record of where each item came from (lineage or provenance), and a version identifier. Data-version tools or simple content hashes let you say exactly which data trained which model — essential when investigating poisoning.

When a model starts behaving oddly, the first question in an investigation is what it learned from. Without versions, the team can only guess whether a suspicious batch of labels was included. With them, it can list every item added since the last good model, find who supplied each, and retrain on a clean version to confirm the cause.

13.2 Integrity of training data

Poisoning requires write access to some training data. Carlini et al. (arXiv:2302.10149) showed that web-scale datasets distributed as URL lists can be poisoned by buying expired domains; integrity hashes of the content at collection time defeat that particular attack. Restrict who can add or relabel data, log changes and review samples from new sources.

Hashes protect what was collected, not whether it was honest in the first place. A labelling contractor with legitimate access can still introduce bad labels, so review samples by source and watch for shifts in label statistics after each delivery.

13.3 Retrieval stores

RAG systems store document chunks and embeddings in a vector database. OWASP LLM 2025 lists vector and embedding weaknesses (LLM08): missing access control lets a user retrieve documents they should not see; poisoned documents can inject instructions. Enforce document-level permissions at retrieval time, using the end user's identity, and label each chunk's provenance and trust level.

Checking with the user's identity matters because the retrieval service itself usually has access to everything. If it searches on its own authority, the model receives documents the user could never open, and a well-chosen question will bring them out in an answer. Provenance labels let the prompt and the policy layer treat text from contractors or outside sources with suitable caution.

13.4 Data protection

Minimise personal data in training and retrieval; set retention limits; consider differential privacy for sensitive training (Abadi et al., arXiv:1607.00133). Membership inference (Shokri et al., arXiv:1610.05820) shows that models can reveal whether a record was used.

Minimisation is the cheapest protection: data never collected cannot leak through a model. Retention limits remove data once its purpose ends, which also shrinks what a breach could expose. Differential privacy adds calibrated noise during training so that no single record changes the model much, at some cost in accuracy. Membership inference makes the risk concrete, since even aggregate outputs can reveal that a particular person's record was used.

13.5 Worked example (synthetic NRS case)

Wrench's vector store held manuals, technician notes and HR safety-incident reports in one index. Any technician could retrieve HR reports through clever questions. Sam split indexes by sensitivity, added per-document access-control lists checked against the requesting user at query time, and added a provenance field (internal/contractor/external) that the prompt template uses to mark untrusted text.

Splitting indexes alone would not have been enough, since a technician querying the right index still needs a per-document check. The provenance field also supports investigations: an answer that quoted contractor text can be traced to its source document.

13.6 Errors and limits

A single index forces one permission model on documents with very different sensitivity, so either access is too broad or the system becomes unusable. Checking with the service identity hands every user the service's rights. Missing training records make poisoning investigations guesswork. Sam's design review asks three matching questions: how many indexes and why, whose identity is checked at retrieval, and where the training-data version is recorded for each model.

Exercises

Synthesis

Data is the attack surface that trains and informs models. Lineage, write control, per-user retrieval permissions and data minimisation protect it.

Continue studying — chapter 13 of 50

← Chapter 12 (draft) · Course chapter index · Chapter 14 (draft) →