◇ OpenCyber Academy
Technologist · Chapter 17 · English

Chapter 17 — AI and machine-learning security: tooling

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 17 — AI and machine-learning security: tooling

Lesson map

Lesson 17 published. Map, example, EX049.

flowchart TD
  N0["Learning outcomes"]
  N1["17.1 Evaluation harness"]
  N2["17.2 Tracing"]
  N3["17.3 Scanners"]
  N4["17.4 Choosing tools"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX049"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["17.1 Evaluation harness"]
  N2["17.2 Tracing"]
  N3["17.3 Scanners"]
  N4["17.4 Choosing tools"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX049"]

[Learning outcomes]
    |
    v
[17.1 Evaluation harness]
    |
    v
[17.2 Tracing]
    |
    v
[17.3 Scanners]
    |
    v
[17.4 Choosing tools]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX049]

Learning outcomes

You will (1) build a small evaluation harness; (2) use logging and tracing tools for AI requests; (3) integrate scanners for dependencies and model files; and (4) choose tools critically.

17.1 Evaluation harness

A harness loads test cases, sets up the environment (plant documents, create test users), runs conversations against a staging deployment, collects tool logs and outputs, grades them with code, and writes a dated report with rates and intervals. It should run from one command and in continuous integration. Keep the harness in version control alongside the application so a change to either can be tested together. Jonas's Ask Brightwell harness is a working example: it plants documents, runs the 120 cases nightly and reports rates with intervals.

17.2 Tracing

Tracing records each step of a request with a shared ID: user message, retrieved chunks, prompt (or its hash and version), model response, tool calls, final answer. Traces make incidents and test failures explainable. Redact personal data, store traces with access control, and set retention limits so the trace store does not become the largest pile of personal data in the company. Ask Brightwell's trace IDs proved their value during the operations tabletop, when the on-call engineer found the injected policy page from a trace in minutes. The earlier shared log bucket shows the risk: prompts with customer addresses readable by 40 engineers. Traces now store the prompt's hash and version, with personal data redacted.

17.3 Scanners

Dependency scanners check libraries for known vulnerabilities; for AI also check model files (hash pinning, format checks that reject code-executing formats) and dataset sources. Coding assistants can suggest packages that do not exist — register-squatting risk — so check new dependencies against an allow-list. A software bill of materials that includes models and datasets helps answer "are we affected?" when an advisory appears. For CodePilot, which suggests packages, the allow-list check matters because a suggested package name might not exist yet and could be registered by an attacker. Brightwell's pipeline also checks model file hashes and rejects pickle-based formats.

17.4 Choosing tools

Many products claim to "secure AI". Ask: what threat does it address? Is it in the request path (adds latency, can block) or monitoring only? How was it evaluated — adaptively? What data does it send to the vendor? Can you test it on your own suites before buying? A tool should fill a gap in your architecture, not replace one. The prompt firewall demonstration showed why the last question matters: on Jonas's held-out suite it blocked many indirect cases but also wrongly stopped some normal questions, a cost the vendor's demonstration did not show. He recorded both rates before any purchase decision, and the product owner weighed them against the policy service that was already in place.

17.5 Worked example (synthetic Brightwell case)

Jonas writes a 300-line Python harness for Ask Brightwell's 120 cases. It runs nightly on staging and posts a report: attack success by category with Wilson intervals and normal-task completion. He adds trace IDs to the orchestrator and a CI check that rejects new dependencies not on the allow-list. A vendor demonstrates a "prompt firewall"; Jonas runs it on his held-out suite before any purchase: it blocks 60% of indirect cases but also 7% of normal questions. The firewall's result is the kind of evidence a purchase decision needs: a gain on attacks and a measured cost to customers, both on Brightwell's own cases.

17.6 Errors and limits

Jonas's work at Brightwell avoids all three. The harness runs from one command nightly, traces redact personal data, and the prompt firewall was tested on the held-out suite before purchase. Testing by hand would have made the later prompt-change regression easy to miss. A vendor demonstration on the vendor's own examples is not a substitute for that test. Each habit feeds evidence into the release gate.

Exercises

Synthesis

Tooling makes AI-security work repeatable: automated harnesses, traces, scanners and tools chosen by testing them against your own threats.

Continue studying — chapter 17 of 30

← Chapter 16 (draft) · Course chapter index · Chapter 18 (draft) →