OpenCyber Academy
Postgraduate · Chapter 7 · English
Course code: OCA-158Chapter 07: OCA-158-C007

Chapter 7 — AI and machine-learning security: organizational integration

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 07 published. Map, example, EX031.

flowchart TD
  N0["Learning outcomes"]
  N1["7.1 Inventory and tiers"]
  N2["7.2 Reuse existing processes"]
  N3["7.3 Roles"]
  N4["7.4 Metrics"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX031"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["7.1 Inventory and tiers"]
  N2["7.2 Reuse existing processes"]
  N3["7.3 Roles"]
  N4["7.4 Metrics"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX031"]

[Learning outcomes]
    |
    v
[7.1 Inventory and tiers]
    |
    v
[7.2 Reuse existing processes]
    |
    v
[7.3 Roles]
    |
    v
[7.4 Metrics]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX031]

Learning outcomes

You will (1) build an AI-system inventory with risk tiers; (2) integrate AI controls into existing security processes (change, access, vendor, incident); (3) define roles and accountability; (4) set organisation-wide metrics; and (5) plan a programme roadmap.

7.1 Inventory and tiers

You cannot secure what you do not know. An inventory lists each AI system, owner, purpose, model source, data, tools, users, affected parties and risk tier. Tier by consequence: what is the worst plausible harm, and how many people does it reach? Agents with write tools and systems affecting rights rank highest. Shadow AI — staff pasting data into public chatbots — belongs in the inventory as a use, even if no system was built.

"""Lab V: an AI system inventory scored for review priority (synthetic).
Each system lists exposure, autonomy, data sensitivity and whether it reads
untrusted content; mapping to OWASP LLM Top 10 (2025) IDs is illustrative."""
import json
inv = json.loads("""[
 {"name":"support-chatbot","exposure":"public","autonomy":"tools","data":"customer","untrusted_input":true},
 {"name":"invoice-ocr-model","exposure":"internal","autonomy":"none","data":"financial","untrusted_input":true},
 {"name":"code-assistant","exposure":"internal","autonomy":"suggest","data":"source","untrusted_input":true},
 {"name":"churn-predictor","exposure":"internal","autonomy":"none","data":"customer","untrusted_input":false},
 {"name":"hr-cv-screener","exposure":"internal","autonomy":"decide","data":"personal","untrusted_input":true}
]""")
E = {"public": 3, "internal": 1}; A = {"none": 0, "suggest": 1, "tools": 3, "decide": 3}
D = {"customer": 2, "financial": 2, "source": 2, "personal": 3}
for s in inv:
    s["score"] = E[s["exposure"]] + A[s["autonomy"]] + D[s["data"]] + (2 if s["untrusted_input"] else 0)
    risks = []
    if s["untrusted_input"]: risks.append("LLM01")
    if s["autonomy"] in ("tools", "decide"): risks.append("LLM06")
    if s["data"] in ("customer", "personal"): risks.append("LLM02")
    s["owasp"] = ",".join(risks) or "-"
for s in sorted(inv, key=lambda s: -s["score"]):
    print(f"{s['name']:18s} score={s['score']:2d} owasp={s['owasp']}")

Expected output:

support-chatbot    score=10 owasp=LLM01,LLM06,LLM02
hr-cv-screener     score= 9 owasp=LLM01,LLM06,LLM02
code-assistant     score= 6 owasp=LLM01
invoice-ocr-model  score= 5 owasp=LLM01
churn-predictor    score= 3 owasp=LLM02

Amira's inventory in the worked example below found 23 AI uses.

7.2 Reuse existing processes

Change management covers model, prompt and retrieval-index changes as well as code. Access management covers the identities agents use. Vendor management asks model and API providers about data use, retention, security testing and incident notification. Incident response adds playbooks for prompt injection, data leakage through a model and poisoned data, including rollback of models and prompts. Integration beats a separate "AI process" that teams can ignore.

At Corvane, Amira added "prompt or model change" to the change-advisory checklist and AI questions to the vendor questionnaire, rather than creating a separate process.

7.3 Roles

Assign: a system owner accountable for each AI system; an AI-security function setting standards and testing; data owners approving data use; and a risk committee accepting residual risk for high tiers. ISO/IEC 42001 expects such roles to be defined and resourced.

At Corvane, Amira's team is the AI-security function, the finance director accepted FraudScore's residual risk, and each built system has a named owner. Write the roles into the inventory so each entry shows who is accountable, and review them when people move.

7.4 Metrics

Useful organisation metrics: percentage of AI systems inventoried and tiered; high-tier systems with current adversarial evaluation; time to roll back a model or prompt; AI-related incidents and near misses; percentage of agent tools with external enforcement. Avoid vanity metrics like "number of prompts filtered".

At Corvane, the first-year roadmap measures the share of high-tier systems evaluated within six months. Report each metric with its trend and say which ones are below target and why, so the board sees where effort is needed.

7.5 Worked example (synthetic Corvane case)

Amira's inventory finds 23 AI uses: 3 built systems, 11 vendor features switched on in SaaS products, and 9 shadow uses. Two vendor features — an email assistant with mailbox access and a CRM summariser — had never been reviewed. She tiers them, adds AI questions to the vendor questionnaire, adds "prompt or model change" to the change-advisory checklist, and writes an injection playbook. Her first-year roadmap targets all high-tier systems evaluated within six months.

The two unreviewed vendor features would have been missed by an inventory of built systems alone.

7.6 Errors and limits

Each error leaves risk unseen or unmanaged. An inventory of built systems alone would have missed Corvane's 11 vendor features and 9 shadow uses. A parallel AI process is easy for teams to ignore; adding questions to existing checklists reaches them. Counting activity, such as prompts filtered, says nothing about whether risk fell; track the share of high-tier systems with current adversarial evaluation instead. Check your programme for all three.

Exercises

Synthesis

Organisational integration means knowing every AI use, folding AI into processes people already follow, assigning accountability and measuring risk reduction.

Continue studying — chapter 07 of 12

← Chapter 06 (draft) · Course chapter index · Chapter 08 (draft) →

Worked answers

EX031. - System: DocRead. Owner: operations IT. Purpose: extract fields from bills of lading. - Model: vendor OCR plus a fine-tuned extractor. Data: scanned documents containing names and addresses. Tools: none, but its outputs feed Copilot and billing. - Users: operations staff. Affected: shippers and consignees. - Tier: medium, because errors propagate into billing and routing but no direct actions are taken.

EX032. The support chatbot takes untrusted user and document input, has tools, and holds personal data, so it maps to LLM01, LLM06 and LLM02. The churn predictor has no tools and limited exposure.

EX033. Model version updates, system-prompt edits, retrieval-index source changes and adding or changing tools or their permissions.

EX034. 1. Is our data used for training, and can we opt out? 2. What is the retention period? 3. What adversarial testing do you perform, and can we see a summary? 4. How fast will you notify us of incidents affecting our data? 5. Can we pin model versions and receive change notices?

EX035. Use: percentage of high-tier systems with evaluation under six months old; time to roll back a prompt or model; percentage of agent tools with external enforcement. Avoid: the count of blocked prompts, which measures activity rather than reduced risk.