Chapter 11 — AI and machine-learning security: assets and context
Lesson map
Lesson 11 published. Map, example, EX031.
flowchart TD
N0["Learning outcomes"]
N1["11.1 AI assets"]
N2["11.2 Context"]
N3["11.3 Ranking"]
N4["11.4 Keeping it current"]
N5["11.4b Inventory fields"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX031"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["11.1 AI assets"]
N2["11.2 Context"]
N3["11.3 Ranking"]
N4["11.4 Keeping it current"]
N5["11.4b Inventory fields"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX031"]
[Learning outcomes]
|
v
[11.1 AI assets]
|
v
[11.2 Context]
|
v
[11.3 Ranking]
|
v
[11.4 Keeping it current]
|
v
[11.4b Inventory fields]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX031]
Learning outcomes
You will (1) inventory AI assets — models, prompts, data, indexes, tools, keys; (2) record context — users, affected people, purpose; (3) rank systems by risk; and (4) keep the inventory current.
11.1 AI assets
Assets include: model files or vendor model identifiers and versions; system prompts; training and evaluation data; retrieval indexes and their sources; tools and their permissions; API keys; logs. Each needs an owner and a protection level. System prompts are not secrets you can rely on — OWASP LLM07 notes that they can leak — so never put credentials in them. Ask Brightwell's inventory entry showed why this matters: its system prompt held a staff discount code, a secret that a leak of the prompt would have exposed.
11.2 Context
The same model is low-risk in a product-search box and high-risk when it can issue refunds. Record purpose, users, affected people, decisions influenced, and tools available. For Brightwell, the same vendor model powers Ask Brightwell and the marketing email writer, but the first can start returns and look up orders, while the second can send mail to the customer list. Their inventory entries record those different purposes and tools, so each gets its own risk score and owner. The tools and data around a model, not the model, set its risk.
11.3 Ranking
Score systems on: untrusted input, tools with side effects, sensitive data, number of affected people, and automation without human review.
"""Lab V: an AI system inventory scored for review priority (synthetic).
Each system lists exposure, autonomy, data sensitivity and whether it reads
untrusted content; mapping to OWASP LLM Top 10 (2025) IDs is illustrative."""
import json
inv = json.loads("""[
{"name":"support-chatbot","exposure":"public","autonomy":"tools","data":"customer","untrusted_input":true},
{"name":"invoice-ocr-model","exposure":"internal","autonomy":"none","data":"financial","untrusted_input":true},
{"name":"code-assistant","exposure":"internal","autonomy":"suggest","data":"source","untrusted_input":true},
{"name":"churn-predictor","exposure":"internal","autonomy":"none","data":"customer","untrusted_input":false},
{"name":"hr-cv-screener","exposure":"internal","autonomy":"decide","data":"personal","untrusted_input":true}
]""")
E = {"public": 3, "internal": 1}; A = {"none": 0, "suggest": 1, "tools": 3, "decide": 3}
D = {"customer": 2, "financial": 2, "source": 2, "personal": 3}
for s in inv:
s["score"] = E[s["exposure"]] + A[s["autonomy"]] + D[s["data"]] + (2 if s["untrusted_input"] else 0)
risks = []
if s["untrusted_input"]: risks.append("LLM01")
if s["autonomy"] in ("tools", "decide"): risks.append("LLM06")
if s["data"] in ("customer", "personal"): risks.append("LLM02")
s["owasp"] = ",".join(risks) or "-"
for s in sorted(inv, key=lambda s: -s["score"]):
print(f"{s['name']:18s} score={s['score']:2d} owasp={s['owasp']}")
Expected output:
support-chatbot score=10 owasp=LLM01,LLM06,LLM02
hr-cv-screener score= 9 owasp=LLM01,LLM06,LLM02
code-assistant score= 6 owasp=LLM01
invoice-ocr-model score= 5 owasp=LLM01
churn-predictor score= 3 owasp=LLM02
Jonas's lab scores Brightwell's systems on these five factors. Ask Brightwell ranks highest because it reads untrusted reviews, has tools that start returns, and handles customer data. The email writer he found later ranks just below it because it has personal data and sends mail. Writing down each factor's score lets owners see why a system ranks where it does and argue with a specific factor rather than the total.
11.4 Keeping it current
Add inventory checks to change management: new model, new tool, new data source or new prompt → update the entry. Scan for shadow AI: vendor features turned on, staff using public chatbots with company data. Brightwell found its fourth AI system, the marketing email writer, only by looking beyond the systems engineering had listed. The customer-service agents pasting emails into a public chatbot, found later during a placement, were shadow use that no system scan could detect. Jonas therefore added a question to each quarterly inventory review asking every team which AI features or chatbots they use with company data.
11.4b Inventory fields
A practical entry has: system name; owner; purpose; model (vendor and version, or file and hash); prompts (location, version); data sources and their sensitivity; indexes; tools and their permissions; keys and where they are stored; users; affected people; risk score; last evaluation date. Store entries in version control so changes are reviewed. Ask Brightwell's entry now records the vendor and pinned model version, the prompt file and its version, the customer and staff indexes, the order and returns tools with their scopes, and where the vendor key is kept.
11.5 Worked example (synthetic Brightwell case)
Jonas builds Brightwell's inventory and finds a fourth system no one listed: the marketing team's AI email writer with access to the customer list. He scores it with the lab's approach; it ranks just below Ask Brightwell because it has personal data and sends mail. He also finds that Ask Brightwell's system prompt contained a discount code meant for staff and moves it out. Both findings came from filling in inventory fields carefully. Asking who uses the customer list revealed the email writer; reading the system prompt to record its location revealed the discount code.
11.6 Errors and limits
- Inventories listing only models.
- Secrets in system prompts.
- One-off inventories.
Brightwell's first inventory showed all three. It listed models but not the email writer's access to the customer list, Ask Brightwell's prompt held a staff discount code, and nothing scheduled a recheck. Version-controlled entries reviewed with each change now address the third, and the risk ranking is rerun whenever an entry changes. Secrets now live only in the secrets manager, so a leaked prompt reveals instructions but nothing that grants access.
Exercises
- EX031 — Assets. List seven asset types for Ask Brightwell with an owner each. Rubric: 3 = seven with owners; 2 = five; 1 = three; 0 = fewer.
- EX032 — Ranking. Explain the lab's ranking of support-chatbot over churn-predictor. Rubric: 3 = untrusted input, tools, data; 2 = two; 1 = one; 0 = wrong.
- EX033 — Prompt secret. Why should the discount code not be in the system prompt? Rubric: 3 = prompts can leak (LLM07), access control belongs elsewhere; 2 = partial; 1 = vague; 0 = wrong.
Synthesis
Know your AI assets and the context they run in, rank by consequence and keep the record current through change management.
Continue studying — chapter 11 of 30
← Chapter 10 (draft) · Course chapter index · Chapter 12 (draft) →