Chapter 22 — AI and machine-learning security: system and data models
Lesson map
Lesson 22 published. Map, example, EX064.
flowchart TD
N0["Learning outcomes"]
N1["22.1 Data-flow diagrams for AI"]
N2["22.2 Provenance and taint"]
N3["22.3 Assets, entry points, sinks"]
N4["22.4 Data models"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX064"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["22.1 Data-flow diagrams for AI"]
N2["22.2 Provenance and taint"]
N3["22.3 Assets, entry points, sinks"]
N4["22.4 Data models"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX064"]
[Learning outcomes]
|
v
[22.1 Data-flow diagrams for AI]
|
v
[22.2 Provenance and taint]
|
v
[22.3 Assets, entry points, sinks]
|
v
[22.4 Data models]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX064]
Learning outcomes
You will (1) build data-flow diagrams for AI systems with trust levels; (2) model data provenance and taint; (3) identify assets, entry points and privileged sinks; and (4) use the models as inputs to threat analysis.
22.1 Data-flow diagrams for AI
Draw external entities (users, suppliers), processes (ingestion, model, policy, tools), data stores (training set, vector store, logs) and flows. Annotate each flow with trust level and data classification. AI adds new elements: training pipelines, model registries, prompt templates and evaluation stores.
Annotations do the analytical work. A flow labelled 'supplier bulletin, external, unclassified' into a process labelled 'model, can propose work orders' is visibly a path from low trust to high impact. Prompt templates belong on the diagram as stores because whoever edits them changes system behaviour, and model registries do too, since a swapped model changes every downstream decision.
22.2 Provenance and taint
Taint tracking marks data from untrusted sources and follows it through processing. In LLM systems, any text derived from untrusted input — including a model's summary of an untrusted document — remains tainted. Tool calls whose arguments are tainted deserve extra checks. Greshake et al. (arXiv:2302.12173) demonstrated how injected content travels from a retrieved source into actions.
Summaries feel cleaner than their source, but an instruction can survive summarising, or the summary itself can be steered by the source text. Wrench therefore carries a taint flag from each retrieved chunk through to any proposal built from it, and the policy service requires confirmation, with the source shown, whenever a tool argument depends on tainted text.
22.3 Assets, entry points, sinks
Assets: data, weights, decisions, credentials, reputation. Entry points: anywhere data enters (uploads, retrieval, sensors, feedback). Privileged sinks: actions with impact (sending e-mail, writing records, actuating equipment). The most important paths run from low-trust entry points to high-impact sinks.
Listing the three separately prevents a common blind spot: entry points that do not look like inputs. Feedback buttons, sensor readings and documents uploaded by colleagues all enter the system. Sinks are listed by impact, so the analysis spends most time on paths that end in physical or financial consequences. For each such path, the team names the control that stands between the entry point and the sink, or records that none exists yet.
22.4 Data models
Describe training data composition (sources, classes, conditions), its update process and who can modify it. A datasheet (Gebru et al., arXiv:1803.09010) is the documentation form; the data model is the engineering form.
The data model states facts engineers can check and act on: what share of TrackEye's images are dawn runs, which supplier contributed each batch, how relabels are approved and who holds write access to the store. When the dawn gap appeared, the data model showed immediately that dawn images were scarce, pointing to the fix. It also defines the update process, so a new source cannot be added without passing provenance and sampling checks.
22.5 Worked example (synthetic NRS case)
Sam's DFD for Wrench marked supplier bulletins and contractor notes as low trust, manuals as medium (they come from manufacturers but are rarely altered), and technician chat as authenticated. Taint propagated through the model's summaries into the create_work_order arguments. Three privileged sinks were found: work-order creation, parts e-mail and a forgotten "update asset status" tool that a pilot had enabled. Sam removed the third tool until it had its own review.
The forgotten tool shows why diagrams are built from configuration, not memory.
22.6 Errors and limits
- Diagrams without trust levels.
- Treating model output as clean because the model "processed" it.
- Missing forgotten tools and data stores.
A diagram without trust levels shows plumbing but not risk. Treating model output as clean ignores that the model can repeat or follow what it read. Forgotten tools and stores are the most dangerous omissions, because nobody reviews controls for components they do not know exist. Sam builds diagrams from the deployed configuration and tool registry, then compares them with the team's own sketch; the differences are usually where the findings are.
Exercises
- EX064 — Taint. Why is a model's summary of an untrusted document still tainted? Rubric: 3 = summary may carry or act on injected instructions/content; 2 = partial; 1 = vague; 0 = wrong.
- EX065 — Sinks. List three privileged sinks in an LLM e-mail assistant. Rubric: 3 = send, delete/move, forward/calendar or contact changes; 2 = two; 1 = one; 0 = none.
- EX066 — DFD. Describe a DFD for TrackEye's training pipeline with trust levels. Rubric: 3 = entities, processes, stores, flows with trust annotations; 2 = missing trust; 1 = vague; 0 = none.
Synthesis
System and data models show where untrusted data enters and where it can cause harm. Taint and trust annotations make the critical paths visible for threat analysis.
Continue studying — chapter 22 of 50
← Chapter 21 (draft) · Course chapter index · Chapter 23 (draft) →