OpenCyber AcademyLesson map
Lesson 01 published. Map, example, EX001.
flowchart TD
N0["Learning outcomes"]
N1["1.1 Layers"]
N2["1.2 Where models live"]
N3["1.3 Resources"]
N4["1.4 Layers and controls"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX001"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["1.1 Layers"]
N2["1.2 Where models live"]
N3["1.3 Resources"]
N4["1.4 Layers and controls"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX001"]
[Learning outcomes]
|
v
[1.1 Layers]
|
v
[1.2 Where models live]
|
v
[1.3 Resources]
|
v
[1.4 Layers and controls]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX001]
Learning outcomes
You will (1) describe the hardware and software layers an AI service runs on; (2) explain where models, weights and data live at run time; (3) identify the computing resources an attacker can exhaust; and (4) relate system layers to AI-security controls.
1.1 Layers
A shop assistant like Ask Brightwell is not "just a model". A request travels through a browser, a load balancer, a web application, an orchestration service that builds the prompt, a retrieval service, a model server (often a vendor API), and tool services such as the order database. Each layer runs on an operating system, on physical or virtual hardware, and each has its own accounts, logs and failure modes. Security work starts by drawing these layers. Jonas's first drawing of Ask Brightwell found a problem in a layer nobody thought of as AI: the orchestrator's logs.
1.2 Where models live
A model is a file of numbers (weights) plus code that runs it. When self-hosted, weights sit on disk, load into memory, and usually run on GPUs. When accessed through an API, the vendor holds them and you send text. Either way, the prompt, retrieved documents and outputs pass through memory and logs on systems you operate — places where sensitive data can leak. For Ask Brightwell the weights stay with the vendor, but the prompts built by the orchestrator, including customer addresses from the order tool, sat in a shared log bucket. CodePilot's self-hosted model files, by contrast, live on Brightwell's own disks, where integrity checks are Brightwell's job.
1.3 Resources
Language-model inference is expensive: each request uses GPU time and memory proportional to the length of input and output. Attackers who send very long or very many requests can raise costs or deny service; OWASP lists this as LLM10 Unbounded Consumption. Knowing which resource is scarce — GPU, tokens, money — tells you what to limit. For Ask Brightwell, the scarce resource is money: each token sent to the vendor API is billed, so an unlimited endpoint lets anyone run up the bill. That is why Jonas's first proposal paired a request rate limit with a daily token budget per account.
1.4 Layers and controls
Map controls to layers: authentication at the edge, rate limits at the gateway, prompt construction and output handling in the orchestrator, access control in retrieval and tools, integrity checks on model files on disk, and logs everywhere. For Ask Brightwell, Jonas's diagram shows where each control sits today and where one is missing. There was no rate limit at the gateway and the orchestrator's prompt logs were readable by 40 engineers, so two layers lacked controls that the map makes obvious. The policy service added later belongs in the same orchestrator layer, between the model and the tools.
1.5 Worked example (synthetic Brightwell case)
Jonas draws Ask Brightwell: browser → CDN → web app → orchestrator → (retrieval index of 3,000 policy and product pages; vendor model API; order-lookup and returns tools) → databases. He notes that the orchestrator logs full prompts, including customer addresses fetched by the order tool, to a shared log bucket that 40 engineers can read. He adds this to the team's issue list. The finding is not about the model at all: it is an ordinary access-control problem in the logging layer. Because the logs hold addresses that the order tool fetched, the bucket became one of the largest stores of customer data in the system without anyone deciding it should be.
1.6 Errors and limits
- Treating the model as the whole system.
- Forgetting logs and caches as data stores.
- Ignoring cost as an attack target.
Jonas's drawing guards against all three. It shows the model as one box among many, lists the shared log bucket as a data store holding customer addresses, and marks the vendor API as a cost to protect. Each error, if left unchecked, would have kept a real Ask Brightwell weakness out of the issue list. Later chapters return to each layer with tests and controls.
Exercises
- EX001 — Layer diagram. Draw the layers of an AI chatbot you use and label one control per layer. Rubric: 3 = at least six layers with apt controls; 2 = four or five; 1 = two or three; 0 = none.
- EX002 — Where data lives. List four places a customer address could be stored during one Ask Brightwell request. Rubric: 3 = four plausible places; 2 = three; 1 = one or two; 0 = none.
- EX003 — Scarce resource. For a self-hosted model, which resource would you limit first and how? Rubric: 3 = GPU time or tokens, per-user limits and max lengths; 2 = partial; 1 = vague; 0 = none.
Synthesis
AI services are layered systems. Security begins by mapping where requests, models and data travel, and which resources attackers can exhaust.