Chapter 19 — Identity and cryptography | Engineering AI and machine-learning security

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 19 — Identity and cryptography

Lesson map

Lesson 19 published. Map, example, EX055.

flowchart TD
  N0["Learning outcomes"]
  N1["19.1 Identities for agents"]
  N2["19.2 Hashing and manifests"]
  N3["19.3 Signatures and provenance"]
  N4["19.4 Encryption"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX055"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["19.1 Identities for agents"]
  N2["19.2 Hashing and manifests"]
  N3["19.3 Signatures and provenance"]
  N4["19.4 Encryption"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX055"]

[Learning outcomes]
    |
    v
[19.1 Identities for agents]
    |
    v
[19.2 Hashing and manifests]
    |
    v
[19.3 Signatures and provenance]
    |
    v
[19.4 Encryption]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX055]

Learning outcomes

You will (1) design identity and access for users, services and AI agents; (2) apply hashing, signatures and encryption to AI artefacts; (3) verify model and dataset integrity with manifests; and (4) explain provenance and signing for the AI supply chain.

19.1 Identities for agents

Every action must be attributable. An AI agent should act on behalf of a user with delegated, scoped permissions (for example OAuth tokens restricted to specific scopes) rather than with a powerful service account. Short-lived credentials limit damage if leaked; never place credentials in prompts.

Delegation means the agent can never do more than the user could, and logs show both: the user on whose behalf the action ran and the agent that proposed it. A shared service account erases that distinction, so after an incident nobody can tell which conversation caused which action. Tokens lasting minutes rather than months also mean a credential copied out of a log is useless by the time anyone tries it.

19.2 Hashing and manifests

A cryptographic hash such as SHA-256 maps a file to a fixed-size digest; any change — even one bit — produces a different digest. A manifest lists the expected digest of every artefact (weights, tokenizer, configuration). The lab detects both a one-bit change in weights and an added configuration line.

"""Lab H: integrity of model artefacts with SHA-256 manifests (stdlib only).
Creates synthetic 'artefact' bytes, records a manifest, then detects a
one-byte tampering and a swapped file. No real models are involved."""
import hashlib, json
files = {"model.bin": bytes(range(256)) * 40, "tokenizer.json": b'{"vocab": ["a", "b"]}',
         "config.yaml": b"layers: 4\nheads: 8\n"}
manifest = {n: hashlib.sha256(b).hexdigest() for n, b in files.items()}
print(json.dumps({k: v[:16] for k, v in manifest.items()}, indent=1))
received = dict(files)
tampered = bytearray(received["model.bin"]); tampered[5000] ^= 1
received["model.bin"] = bytes(tampered)
received["config.yaml"] = b"layers: 4\nheads: 8\ntrust_remote_code: true\n"
for n, b in sorted(received.items()):
    ok = hashlib.sha256(b).hexdigest() == manifest[n]
    print(f"{n:15s} {'OK' if ok else 'MISMATCH'}")

Expected output:

{
 "model.bin": "e96760a87768717b",
 "tokenizer.json": "f63d2ddabc61f2d9",
 "config.yaml": "e5bbb570ca0fcb87"
}
config.yaml     MISMATCH
model.bin       MISMATCH
tokenizer.json  OK

The output shows why every file needs its own entry: the tokenizer was untouched and passes, while two different changes in two files are each caught independently. A single flipped bit at offset 5000 is enough to change the weight file's digest completely.

19.3 Signatures and provenance

A hash proves a file matches a manifest; a digital signature on the manifest proves who produced it. Signing model artefacts and recording provenance (which data, code and pipeline produced them) lets consumers verify origin. This addresses OWASP LLM03 (supply chain) and backdoor risks from third-party models (Gu et al., arXiv:1708.06733).

Signatures move the trust question to the signing key: who holds it, how it is protected and how a consumer learns which key to expect. Provenance adds context that a signature alone cannot give, such as the dataset version used, so a consumer can judge whether the model is fit for their use and not merely genuine.

19.4 Encryption

Encrypt training data and weights at rest and all traffic in transit (TLS). Encryption protects confidentiality against storage theft and interception but not against a legitimate-looking query that extracts information from the model.

Keys must be managed separately from the data they protect; an encrypted weight file stored next to its key protects nothing. Encryption also leaves the model's own behaviour untouched. A user with ordinary query access can still extract training data or reconstruct a model through its answers, so encryption is paired with query limits and output controls. Engineers record which threats each encryption control addresses and which it explicitly does not.

19.5 Worked example (synthetic NRS case)

NRS receives TrackEye model updates from a vision-model supplier. Sam required a signed manifest with SHA-256 digests for each file, verified on the inspection-train computer before loading. In a test, a modified configuration enabling remote code loading was rejected, matching the lab's config.yaml MISMATCH.

Verifying on the train rather than in the depot matters because the files could be altered anywhere between the supplier and the device. The manifest's signature is checked first; the file hashes are trusted only once the signature is shown to come from the supplier's expected key.

19.6 Errors and limits

Agents using shared admin service accounts.
Hashes fetched from the same place as the files (an attacker can change both).
Assuming encryption prevents model-level leakage.

Shared admin accounts make every agent action as powerful as the most powerful task any agent needs. A hash downloaded from the same server as the file proves only that the server is consistent with itself, so the expected digest must arrive through a separate, trusted channel such as a signed manifest. Encryption protects data at rest and in transit, but a model answering queries can still reveal what it learned.

Exercises

EX055 — Lab reading. Which files fail in Lab H and why does tokenizer.json pass? Rubric: 3 = model.bin (bit flip) and config.yaml (added line); tokenizer unchanged; 2 = partial; 1 = vague; 0 = wrong.
EX056 — Same-source hashes. Why is a manifest downloaded from the same server as the model weak, and what fixes it? Rubric: 3 = attacker who controls server changes both; signed manifest with pinned key; 2 = partial; 1 = vague; 0 = wrong.
EX057 — Delegation. Design the credential for Wrench's work-order tool. Rubric: 3 = per-user delegated, scoped to create-draft, short-lived, logged; 2 = three; 1 = one; 0 = none.

Synthesis

Identity makes agent actions attributable and bounded; hashes, signatures and provenance make AI artefacts verifiable. Together they anchor trust in the AI supply chain.

Continue studying — chapter 19 of 50

← Chapter 18 (draft) · Course chapter index · Chapter 20 (draft) →