Chapter 15 — AI and machine-learning security: integration and dependencies
Lesson map
Lesson 15 published. Map, example, EX057.
flowchart TD
N0["Learning outcomes"]
N1["15.1 Dependencies"]
N2["15.2 Vendor and open-source checks"]
N3["15.3 Secure integrations"]
N4["15.4 Advisories"]
N5["15.4b Watching for changes"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX057"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["15.1 Dependencies"]
N2["15.2 Vendor and open-source checks"]
N3["15.3 Secure integrations"]
N4["15.4 Advisories"]
N5["15.4b Watching for changes"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX057"]
[Learning outcomes]
|
v
[15.1 Dependencies]
|
v
[15.2 Vendor and open-source checks]
|
v
[15.3 Secure integrations]
|
v
[15.4 Advisories]
|
v
[15.4b Watching for changes]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX057]
Learning outcomes
You will (1) list the dependencies of an AI service; (2) check vendor and open-source components; (3) secure integrations between services; (4) respond to dependency advisories.
15.1 Dependencies
An AI service depends on: the model (vendor API or file), libraries (orchestration, retrieval, parsing), the vector database, tool services, the identity provider and the secrets store. A supply-chain failure in any of them affects the service — OWASP LLM03 Supply Chain.
For HelpBot that means: the vendor model API, the orchestration library, the document-parsing library used during ingestion, the vector database holding the article index, the ticketing and router-management services, the identity provider that authenticates customers, and the secrets store holding the API key and router credential. The document-parsing advisory showed that a library customers never see can still be the weakest point, since it handles every file uploaded to the index.
15.2 Vendor and open-source checks
For model vendors: data use and retention, version pinning, change notice, incident notification. For open-source models and libraries: source, licence, maintenance, known vulnerabilities, file format (avoid formats that execute code on load), and hash pinning. Keep a list of all components and versions (a bill of materials).
Kestrel's bill of materials lists each HelpBot component with its version and image. Its gap was the vendor model, which had no entry because nobody installed it; Ana added the pinned model version and the vendor's change-notice address. For LineWatch's self-hosted model, the check includes the file format: a format that runs code on load, combined with trust_remote_code in the configuration, is the case the hash lab's config.yaml mismatch illustrates.
15.3 Secure integrations
Each integration has a written contract: data exchanged, format, validation, authentication, limits, error handling. Treat data from other services — including other AI services — as untrusted. Fail closed on errors.
HelpBot's contract with the router-management service is an example: the tool service sends a router ID matching R- plus five digits and the customer's identity, over mutual TLS, at most 3 restarts per router per day; any error, timeout or unexpected field means no restart and a logged denial. Failing closed means that when the account database is unreachable, the ownership check refuses rather than allows, as it did for the customer with a missing router ID.
15.4 Advisories
When an advisory is published (a library vulnerability, a vendor incident), check the bill of materials, assess exposure, patch or mitigate, verify with the test suite, and record what was done.
"""Lab V: an AI system inventory scored for review priority (synthetic).
Each system lists exposure, autonomy, data sensitivity and whether it reads
untrusted content; mapping to OWASP LLM Top 10 (2025) IDs is illustrative."""
import json
inv = json.loads("""[
{"name":"support-chatbot","exposure":"public","autonomy":"tools","data":"customer","untrusted_input":true},
{"name":"invoice-ocr-model","exposure":"internal","autonomy":"none","data":"financial","untrusted_input":true},
{"name":"code-assistant","exposure":"internal","autonomy":"suggest","data":"source","untrusted_input":true},
{"name":"churn-predictor","exposure":"internal","autonomy":"none","data":"customer","untrusted_input":false},
{"name":"hr-cv-screener","exposure":"internal","autonomy":"decide","data":"personal","untrusted_input":true}
]""")
E = {"public": 3, "internal": 1}; A = {"none": 0, "suggest": 1, "tools": 3, "decide": 3}
D = {"customer": 2, "financial": 2, "source": 2, "personal": 3}
for s in inv:
s["score"] = E[s["exposure"]] + A[s["autonomy"]] + D[s["data"]] + (2 if s["untrusted_input"] else 0)
risks = []
if s["untrusted_input"]: risks.append("LLM01")
if s["autonomy"] in ("tools", "decide"): risks.append("LLM06")
if s["data"] in ("customer", "personal"): risks.append("LLM02")
s["owasp"] = ",".join(risks) or "-"
for s in sorted(inv, key=lambda s: -s["score"]):
print(f"{s['name']:18s} score={s['score']:2d} owasp={s['owasp']}")
Expected output:
support-chatbot score=10 owasp=LLM01,LLM06,LLM02
hr-cv-screener score= 9 owasp=LLM01,LLM06,LLM02
code-assistant score= 6 owasp=LLM01
invoice-ocr-model score= 5 owasp=LLM01
churn-predictor score= 3 owasp=LLM02
In the document-parsing advisory, those steps took two days. The bill of materials showed HelpBot's ingestion image used an affected version. Exposure was real but narrow: only files entering the index were parsed, so pausing ingestion contained it. The upgrade was rebuilt from the pipeline, the HelpBot suite was run on staging and passed, ingestion resumed, and the record listed the advisory, the versions before and after, and the test results.
15.4b Watching for changes
Subscribe to security advisories for your main libraries and to change notices from your model vendor. When a vendor announces a model update, plan a re-test on staging before the change reaches production, or pin the old version until you have done so.
Kestrel pins HelpBot's vendor model version and receives the vendor's change notices at the service operations mailbox. When a new version is announced, the HelpBot suite runs against it on staging first; the old version stays pinned until the gate passes. This is what keeps "model" a checked cause in troubleshooting rather than a guess.
15.5 Worked example (synthetic Kestrel case)
A popular document-parsing library used by HelpBot's retrieval pipeline has a vulnerability allowing crafted files to run code. Ana checks the bill of materials: version affected. Mitigation the same day: pause ingestion of new documents; next day: upgrade, rebuild, run the suite, resume ingestion. She also adds the vendor model's version to the bill of materials, which had been missing.
Pausing ingestion was chosen because it removed exposure without taking HelpBot offline: customers kept getting answers from the existing index, and only new articles waited a day. The missing model version was found because Ana went through the bill of materials line by line during the advisory, not just the affected library.
15.6 Errors and limits
- No bill of materials.
- Trusting another service's output.
- Loading model files in code-executing formats.
Without a bill of materials, the parsing advisory would have meant checking every image by hand while ingestion stayed open. Trusting another service's output would let a compromised ticketing system feed instructions back into HelpBot through ticket text. Loading model files in code-executing formats turns a swapped LineWatch model into code running inside Kestrel's network, which is why the hash check and the read-only mount sit together.
During the parsing advisory Ana confirmed LineWatch's model file used a format that does not run code on load.
Exercises
- EX057 — Dependencies. List eight dependencies of HelpBot. Rubric: 3 = eight apt; 2 = six; 1 = three; 0 = fewer.
- EX058 — Vendor questions. Write four questions for a model vendor. Rubric: 3 = four apt; 2 = three; 1 = one or two; 0 = none.
- EX059 — Advisory. Write the steps Ana followed as a procedure. Rubric: 3 = check BOM, assess, mitigate, patch, verify, record; 2 = four; 1 = two; 0 = none.
- EX060 — Inventory lab. Why does the lab score some systems higher? Rubric: 3 = untrusted input, tools, data; 2 = two; 1 = one; 0 = wrong.
Synthesis
AI services are only as secure as their dependencies: list them, check them, define integration contracts and respond quickly to advisories.
Continue studying — chapter 15 of 18
← Chapter 14 (draft) · Course chapter index · Chapter 16 (draft) →