◇ OpenCyber Academy
Engineering · Chapter 28 · English

Chapter 28 — AI and machine-learning security: implementation I

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 28 — AI and machine-learning security: implementation I

Lesson map

Lesson 28 published. Map, example, EX082.

flowchart TD
  N0["Learning outcomes"]
  N1["28.1 Policy enforcement point"]
  N2["28.2 Provenance labelling"]
  N3["28.3 Output validation and secret scanning"]
  N4["28.4 Testing the implementation"]
  N5["28.6b Failing closed in the enforcement point"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX082"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["28.1 Policy enforcement point"]
  N2["28.2 Provenance labelling"]
  N3["28.3 Output validation and secret scanning"]
  N4["28.4 Testing the implementation"]
  N5["28.6b Failing closed in the enforcement point"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX082"]

[Learning outcomes]
    |
    v
[28.1 Policy enforcement point]
    |
    v
[28.2 Provenance labelling]
    |
    v
[28.3 Output validation and secret scanning]
    |
    v
[28.4 Testing the implementation]
    |
    v
[28.6b Failing closed in the enforcement point]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX082]

Learning outcomes

You will (1) implement a policy enforcement point for model-proposed actions; (2) implement provenance labelling in prompts; (3) implement output validation and secret scanning; and (4) test the implementation with unit, property and adversarial tests.

28.1 Policy enforcement point

The model emits a proposal (tool name and arguments in a schema). The enforcement point validates the schema, checks the tool against the user's role, checks arguments against scope (depot, asset), checks whether arguments are tainted by untrusted content, and decides: allow, require confirmation, or deny. Decisions are logged with reasons.

ROLE_TOOLS = {"technician": {"search", "create_work_order"}, "viewer": {"search"}}
def decide(role, tool, args, tainted):
    if tool not in ROLE_TOOLS.get(role, set()):
        return "deny: tool not allowed for role"
    if tool == "create_work_order":
        return "confirm: tainted arguments" if tainted else "confirm: high impact"
    return "allow"
for case in [("viewer", "create_work_order", {}, False), ("technician", "search", {}, True),
             ("technician", "create_work_order", {"asset": "AX12"}, True)]:
    print(case[:2], "->", decide(*case))

It prints deny, allow and confirm: tainted arguments for the three cases.

The order of checks matters: role first, so a viewer's request is refused before anything else is examined, then impact and taint. A technician's search with tainted arguments is allowed, because search changes nothing, while any work order needs confirmation and the tainted one says why.

28.2 Provenance labelling

When assembling a prompt, wrap each retrieved chunk with its source and trust level, and state in the system instructions that labelled untrusted content is data. This helps the model but is not a security boundary; the enforcement point remains authoritative.

A typical wrapper reads: '[source: supplier bulletin SB-2291, trust: external] … [end source]'. Labels improve the model's behaviour on average and give technicians context when they review a proposal, since the confirmation screen can show which labelled sources influenced it. But an attacker can write text that imitates a label or tells the model to disregard it, which is why the label informs and the enforcement point decides.

28.3 Output validation and secret scanning

Validate structured outputs with strict parsers. Scan free-text outputs for secrets (API-key patterns, internal hostnames) and for personal data before display or sending. Escape outputs before rendering in web pages to avoid script injection (OWASP LLM05).

A strict parser rejects anything outside the expected structure instead of guessing what was meant. Secret scanning uses the same patterns as code repositories and adds internal names that should never leave the organisation. Escaping matters because a model can be induced to emit script tags or links, and a chat interface that renders them would run an attacker's code in a technician's browser.

28.4 Testing the implementation

Unit tests cover each rule; property tests generate random proposals and check invariants; adversarial tests replay a corpus of injected documents end to end and verify that no forbidden action executes.

The three kinds catch different faults. Unit tests show each rule does what it says. Property tests find combinations nobody thought of, such as an empty role string. End-to-end replay checks the whole path, including adapters and logging, which is where ordering bugs hide. The replay corpus grows with every new attack pattern found, and a case once added is never removed, so old weaknesses cannot quietly return.

28.5 Worked example (synthetic NRS case)

Sam implemented the enforcement point as a separate service with 140 lines of policy code and 60 tests. An adversarial replay of 600 synthetic injected bulletins produced 41 model proposals for work orders; all 41 were routed to confirmation with the tainted flag, and none executed automatically.

Forty-one proposals in 600 is about 6.8%, well above the 2% that the requirement allows for proposals, so the replay also produced a finding for the prompt team. The enforcement point did its own job: the excess proposals reached a technician's confirmation screen marked as tainted, and nothing ran. The 60 tests are part of the build, so any change to the policy code must pass them before release.

28.6 Errors and limits

Rules stated only in the prompt are suggestions to the model, not enforcement. Unescaped output turns the chat window into an attack surface for script injection. Happy-path tests confirm that allowed actions work but never show that forbidden ones are refused. The fail-closed test described next is one example of the negative testing that every rule in the enforcement point needs, alongside replay of injected documents.

28.6b Failing closed in the enforcement point

If Wrench's policy enforcement point cannot load its rules or reach its decision service, it refuses the requested tool action and tells the technician why, rather than allowing it. NRS tests this path in every release by starting the service with a missing policy file.

The test matters because the opposite failure is silent: an enforcement point that allows actions when its policy file is missing looks healthy in every functional test. In Sam's 600-bulletin replay, all 41 work-order proposals reached a technician's confirmation screen; with a fail-open service and no policy loaded, the same 41 would have become work orders with nobody checking them.

Exercises

Synthesis

Implementation makes the architecture real: a small, tested enforcement point, labelled provenance, strict output handling and adversarial replay tests.

Continue studying — chapter 28 of 50

← Chapter 27 (draft) · Course chapter index · Chapter 29 (draft) →