Chapter 2 — Computing systems | Bachelor's AI and machine-learning security

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 02 published. Map, example, EX004.

flowchart TD
  N0["Learning outcomes"]
  N1["2.1 Layers of a computing system"]
  N2["2.2 Where AI runs"]
  N3["2.3 Shared responsibility"]
  N4["2.4 Mapping an AI service"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX004"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["2.1 Layers of a computing system"]
  N2["2.2 Where AI runs"]
  N3["2.3 Shared responsibility"]
  N4["2.4 Mapping an AI service"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX004"]

[Learning outcomes]
    |
    v
[2.1 Layers of a computing system]
    |
    v
[2.2 Where AI runs]
    |
    v
[2.3 Shared responsibility]
    |
    v
[2.4 Mapping an AI service]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX004]

Learning outcomes

You will (1) describe how computers, operating systems, networks and cloud services fit together; (2) identify where an AI application runs and stores data; (3) explain the shared-responsibility model for cloud and hosted AI; and (4) map an AI service to its computing components.

2.1 Layers of a computing system

Hardware runs an operating system, which runs applications; applications talk over networks to other services, often in the cloud. An AI application adds model files, inference servers, vector databases and connections to external model APIs. Each layer has its own owners and failure modes.

For Ask Lumen, the extra parts are the hosted model, the vector database of clinic documents and the booking tool. Each adds owners: the model provider, LCH's data team and the scheduling system's administrators. A security problem in any one, such as a misconfigured database, affects the chatbot even though the model itself works correctly. Knowing the layers tells Rina whom to ask about each risk.

2.2 Where AI runs

A model can run on a user's device, on the organisation's servers, or at a provider via an API. Each choice changes who can see data, who can change the model and who can test it. Hosted AI APIs process prompts on the provider's infrastructure; the organisation controls only its own code, configuration and data sent.

Ask Lumen uses a hosted API, so LCH cannot inspect the model's weights or patch it; it can only choose what to send and how to treat what comes back. That makes LCH's own code, the part that builds prompts and checks outputs, the main place where controls can be added.

2.3 Shared responsibility

In cloud computing, the provider secures the infrastructure and the customer secures what they put on it. For hosted AI, the provider is typically responsible for model hosting and platform security; the customer is responsible for what data they send, how outputs are used, what tools the model can trigger and who can access the application. Misunderstanding this split is a common cause of incidents.

In Rina's map, the booking service account is the clearest example: the model provider has no control over what that account can do in LCH's scheduling system.

2.4 Mapping an AI service

List: user interface; application server; model (where and which version); retrieval store; tools and the systems they touch; logs; identities used at each step. This map is the starting point for every later chapter.

Each item answers a question an attacker or investigator would ask: where does input enter, where does data rest, what can the system change, and under whose identity. A map that lists only the model misses most of the attack surface, because in Ask Lumen the risky actions happen through the booking tool and the retrieval store. Rina keeps her map in a shared document and updates it whenever LCH changes a component.

2.5 Worked example (synthetic LCH case)

Rina mapped Ask Lumen: web chat widget → LCH application server (cloud VM) → hosted language-model API (provider) → retrieval from a vector database of clinic documents (LCH-managed) → booking tool calling the scheduling system (on-premises) using a service account. Logs went to the cloud provider's log service. Rina noted that the booking service account could book for any patient — a responsibility LCH, not the model provider, must fix.

The map also shows who must act on each finding: the service account belongs to LCH's scheduling team, and the fix is to restrict it to the patient whose session made the request.

2.6 Errors and limits

Assuming the model provider secures the whole application.
Forgetting tools and logs in the map.
Not knowing which model version is in use.

Each error hides part of the risk. Assuming the provider secures everything leaves LCH's service account unexamined. Leaving out tools and logs hides the routes by which the chatbot can act on or record patient data. Not knowing the model version makes it impossible to tell whether a change in behaviour came from the provider updating the model or from LCH's own code.

Exercises

EX004 — Responsibility. For Ask Lumen, list two provider and three LCH responsibilities. Rubric: 3 = correct split; 2 = minor error; 1 = confused; 0 = none.
EX005 — Map. Map a school's AI homework helper into the seven components of 2.4. Rubric: 3 = all seven plausible; 2 = five; 1 = three; 0 = none.
EX006 — Deployment choice. Give one security advantage of on-device models and one of hosted APIs. Rubric: 3 = data stays local vs provider-managed patching/scale; 2 = one; 1 = vague; 0 = wrong.

Synthesis

AI applications are computing systems with extra parts. Mapping where each part runs and who is responsible for it is the first security step.

Continue studying — chapter 02 of 40

← Chapter 01 (draft) · Course chapter index · Chapter 03 (draft) →