◇ OpenCyber Academy
Engineering · Chapter 9 · English

Chapter 9 — Programming II

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 9 — Programming II

Lesson map

Lesson 09 published. Map, example, EX025.

flowchart TD
  N0["Learning outcomes"]
  N1["9.1 Modules and interfaces"]
  N2["9.2 Dependencies and supply chain"]
  N3["9.3 Testing beyond examples"]
  N4["9.4 An evaluation harness"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX025"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["9.1 Modules and interfaces"]
  N2["9.2 Dependencies and supply chain"]
  N3["9.3 Testing beyond examples"]
  N4["9.4 An evaluation harness"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX025"]

[Learning outcomes]
    |
    v
[9.1 Modules and interfaces]
    |
    v
[9.2 Dependencies and supply chain]
    |
    v
[9.3 Testing beyond examples]
    |
    v
[9.4 An evaluation harness]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX025]

Learning outcomes

You will (1) structure larger programs into modules with clear interfaces; (2) manage dependencies and their supply-chain risks; (3) write tests, including property-based and regression tests; and (4) build a small evaluation harness for an AI component.

9.1 Modules and interfaces

Large programs are divided into modules with narrow interfaces. For AI systems, separate data loading, model inference, policy enforcement and tool execution. The policy module should be small, reviewable and independent of the model: it decides what is allowed regardless of what the model says. This is the software form of least privilege.

If the policy depends on the model's own judgement, a successful prompt injection also defeats the policy. Keeping it in separate code means it applies the same rules however the model was persuaded. Narrow interfaces also make reviews possible: a reviewer can read an 80-line module carefully, but not a system where checks are spread across prompts and scripts.

9.2 Dependencies and supply chain

AI projects pull many libraries and model files. Pin versions, verify hashes, and prefer safe serialisation: loading untrusted pickle files in Python can execute arbitrary code, which is why formats designed to hold only tensors are preferred for model weights. OWASP LLM 2025 lists supply chain (LLM03) and data and model poisoning (LLM04). Gu et al. (arXiv:1708.06733) showed that a backdoored model obtained from a third party can behave normally on clean inputs.

Pinning stops an upgrade from changing behaviour silently, and hash checks stop a swapped file from being loaded. A backdoored model is harder: it may pass every ordinary test, because its harmful behaviour appears only with a specific trigger. Engineers reduce this risk by preferring sources with published provenance, recording where each model came from, and testing for unexpected behaviour on inputs outside the usual distribution.

9.3 Testing beyond examples

Regression tests freeze known-good behaviour so changes cannot silently break it. Property-based tests generate many inputs and check invariants ("the policy never allows create_work_order without a confirmation flag"). For AI components, a golden set of inputs with expected outputs detects behavioural drift when a model or prompt changes.

Each kind catches something different. Regression tests stop old bugs returning, property tests find inputs nobody thought of, and golden sets show when a prompt edit or a model update changes answers that used to be right. Golden sets need care, because model outputs vary; Sam checks properties of the answers, such as which tool was proposed, rather than exact wording.

9.4 An evaluation harness

A harness loads a test set, runs the component, records outputs and computes metrics with intervals. It must be deterministic given a seed, log versions (model hash, prompt version, code commit) and keep test data separate from development data.

Without logged versions, a result cannot be tied to the system it describes, and a later regression cannot be traced. Determinism makes reruns comparable, so a change in metrics reflects a change in the system rather than in random sampling. Keeping test and development data apart stops the system from being tuned on the very examples used to judge it, which would make results look better than they are.

9.5 Worked example (synthetic NRS case)

Sam restructured Wrench into four modules. The policy module (80 lines) checks every proposed tool call against the user's role and requires a confirmation token for work orders. A property-based test generated 10,000 random tool-call requests and confirmed none passed without a token. The dependency review found a model-conversion script that loaded pickle files from a shared folder; Sam replaced it with a tensor-only format loader and added hash verification.

The property test samples the space of tool calls far more widely than hand-written cases could, though it does not prove that no unsafe call exists.

9.6 Errors and limits

Policy written into a prompt can be talked around by the model's input, while policy in code cannot. Unpinned dependencies make yesterday's test results meaningless today. A golden set used during tuning gradually stops being an independent check, because the system has been adjusted to fit it. Sam keeps a second, untouched set for release decisions and refreshes it when the system's use changes.

Exercises

Synthesis

Good structure puts security decisions in small, testable code outside the model; disciplined dependencies and tests keep that structure trustworthy as the system grows.

Continue studying — chapter 09 of 50

← Chapter 08 (draft) · Course chapter index · Chapter 10 (draft) →