◇ OpenCyber Academy
Engineering · Chapter 39 · English

Chapter 39 — AI and machine-learning security: maintainability

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 39 — AI and machine-learning security: maintainability

Lesson map

Lesson 39 published. Map, example, EX115.

flowchart TD
  N0["Learning outcomes"]
  N1["39.1 Change is constant"]
  N2["39.2 Update strategy"]
  N3["39.3 Keeping knowledge current"]
  N4["39.4 Technical debt in AI security"]
  N5["39.5 Retirement"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX115"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["39.1 Change is constant"]
  N2["39.2 Update strategy"]
  N3["39.3 Keeping knowledge current"]
  N4["39.4 Technical debt in AI security"]
  N5["39.5 Retirement"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX115"]

[Learning outcomes]
    |
    v
[39.1 Change is constant]
    |
    v
[39.2 Update strategy]
    |
    v
[39.3 Keeping knowledge current]
    |
    v
[39.4 Technical debt in AI security]
    |
    v
[39.5 Retirement]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX115]

Learning outcomes

You will (1) design AI systems whose security can be maintained as models, data and threats change; (2) manage model and dependency updates; (3) keep evaluation suites, threat models and documentation current; and (4) plan retirement and data disposal.

39.1 Change is constant

Hosted models are updated; open models gain new versions; libraries patch vulnerabilities; attackers publish new techniques; business processes change. A secure design that cannot be updated safely will become insecure.

Each source of change affects security differently. A model update may alter how readily it follows injected text; a library patch may change parsing behaviour; a new attack technique may defeat a filter that worked last month. Maintainability is the ability to absorb these changes without losing the controls that matter, which needs tests, owners and a predictable process for updates.

39.2 Update strategy

Pin versions and update deliberately: each model or dependency update triggers the security evaluation gate, a canary deployment and a rollback plan. For hosted models that change without notice, run the evaluation suite on a schedule and alert on regressions. Keep abstraction layers so that a model can be swapped without rewriting policy enforcement.

The abstraction layer means policy enforcement does not depend on any particular model's behaviour. Swapping a model then needs only the gate and canary, not a redesign of the controls around it.

39.3 Keeping knowledge current

Assign owners for the threat model, evaluation suites, model cards and runbooks, with review dates. Add every incident and red-team finding to the regression suite. Track external sources (OWASP updates, MITRE ATLAS additions, NIST publications) and record whether each change affects your system.

An owner with a review date turns 'someone should update the threat model' into a task with a name attached. Adding every finding to the regression suite means the system's tests grow from real experience rather than from guesswork. External tracking is recorded briefly, for example 'new ATLAS technique on tool-description injection: applies to connectors, test added', so later reviewers can see that changes were considered.

39.4 Technical debt in AI security

Debt accumulates as one-off filters, exceptions and undocumented prompt tweaks. Periodically refactor: remove controls that no longer serve a threat, consolidate filters into the enforcement point, and document remaining exceptions with expiry dates.

A filter added during an incident and never removed may now block legitimate requests or duplicate a control in the enforcement point. Undocumented prompt tweaks are especially risky, since nobody knows which one is holding back a known weakness. Refactoring is done with the regression suite running, so removing a control that still matters shows up as a failing test rather than an incident.

39.5 Retirement

Retiring a model includes revoking its credentials, removing access paths, deciding what to do with weights (they may encode training data), deleting or archiving data according to retention rules, and updating inventories. ISO/IEC 42001 expects AI system lifecycle processes including retirement.

Forgotten models are a real risk: an old endpoint still running with valid credentials can be found and abused long after everyone stopped using it. Weights may allow membership inference or leak training data, so they are handled under the same rules as the data itself. The inventory update closes the loop, so monitoring and audits stop expecting a system that no longer exists and notice any that should not still be running.

39.6 Worked example (synthetic NRS case)

A hosted embedding model used for Wrench's retrieval was silently updated by the provider. The scheduled evaluation found that retrieval now ranked HR documents higher for some queries; the access-control filter still blocked them, but the team was alerted. Sam added "embedding model version" to monitoring and moved to a pinned version with scheduled upgrades.

Defence in depth held: the ranking change was harmless only because access control was enforced separately from retrieval. The episode also showed a hidden dependency, since nobody had listed the embedding model as something that could change.

39.7 Errors and limits

Without owners, threat models and suites stop changing while the system keeps changing, and the gap widens unseen. Exceptions without expiry accumulate until nobody knows which are still needed. Retirement that removes the interface but leaves weights and credentials behind creates an unmonitored system holding sensitive material. Sam's quarterly review checks owner names, exception dates and the inventory against what is actually running.

Exercises

Synthesis

Maintainable AI security assumes change: pinned and gated updates, owned and current knowledge, managed debt and deliberate retirement.

Continue studying — chapter 39 of 50

← Chapter 38 (draft) · Course chapter index · Chapter 40 (draft) →