◇ OpenCyber Academy
Bachelor's · Chapter 32 · English

Chapter 32 — AI and machine-learning security: interdisciplinary interfaces

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 32 — AI and machine-learning security: interdisciplinary interfaces

Lesson map

Lesson 32 published. Map, example, EX094.

flowchart TD
  N0["Learning outcomes"]
  N1["32.1 Law and regulation"]
  N2["32.2 Management systems and frameworks"]
  N3["32.3 Procurement"]
  N4["32.4 Clinical and domain experts"]
  N5["32.5 Communicating risk"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX094"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["32.1 Law and regulation"]
  N2["32.2 Management systems and frameworks"]
  N3["32.3 Procurement"]
  N4["32.4 Clinical and domain experts"]
  N5["32.5 Communicating risk"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX094"]

[Learning outcomes]
    |
    v
[32.1 Law and regulation]
    |
    v
[32.2 Management systems and frameworks]
    |
    v
[32.3 Procurement]
    |
    v
[32.4 Clinical and domain experts]
    |
    v
[32.5 Communicating risk]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX094]

Learning outcomes

You will (1) work with legal, clinical, privacy, procurement and management colleagues on AI security; (2) interpret regulatory and standards requirements at an orientation level; (3) translate technical risk into organisational decisions; and (4) recognise when expert advice is required.

32.1 Law and regulation

The EU AI Act (Regulation (EU) 2024/1689) sets obligations by risk category, with staged application dates between 2025 and 2027; high-risk systems need risk management, data governance, documentation, logging, human oversight (Article 14) and accuracy, robustness and cybersecurity (Article 15). Data-protection law (such as the GDPR) governs personal data in training, prompts and logs. Health-sector rules may add requirements. Security engineers provide evidence; lawyers decide legal classification.

Rina's role in LCH's assessment was to describe what each system does and what controls exist; the lawyer decided the classification.

32.2 Management systems and frameworks

ISO/IEC 42001:2023 specifies an AI management system: policy, roles, risk assessment, controls, monitoring and improvement. The NIST AI RMF provides Govern, Map, Measure and Manage functions. These frameworks connect security work to organisational accountability.

At LCH, the AI RMF's Map function corresponds to Rina's system models and threat register, and Measure to her verification reports. Using that structure helped the board see that AI security work was part of the clinic's normal risk process. An ISO/IEC 42001 certification was discussed but deferred until the processes had run for a year.

32.3 Procurement

When buying AI products, ask vendors for: model and data provenance, security testing evidence, incident notification commitments, data-use terms (is customer data used for training?), logging and audit support, and the ability to test under agreed terms.

The question about training on customer data matters especially in health care, where conversations contain diagnoses. The ability to test under agreed terms matters because LCH cannot otherwise verify vendor claims, as the broken-defences research showed. Rina keeps these questions in a standard procurement checklist so they are asked for every AI purchase.

32.4 Clinical and domain experts

Domain experts define harm: which wrong answer is dangerous, which delay matters. Security priorities depend on their judgement.

When Rina rated threats for Ask Lumen, a clinician changed her view of impact: a false clinic closure seemed minor to her, but for patients due for time-sensitive treatment, such as wound checks, a missed appointment could cause harm. The impact rating rose, and the grounding check moved up the priority list. Rina now asks a clinician to review impact ratings for every new threat.

32.5 Communicating risk

Translate findings into business terms: impact on patients, legal exposure, cost, reputation, and options with trade-offs. Bring a recommendation, not just a problem.

For the triage tool purchase, Rina's summary to the board had three options: sign as offered, sign with conditions, or choose another vendor. She recommended signing with conditions and stated the residual risk, the vendor's limited test evidence, in plain terms. The board accepted the recommendation because the trade-offs were laid out clearly on one page in plain language.

32.6 Worked example (synthetic LCH case)

LCH planned to buy an AI triage tool. Rina joined the procurement panel with the data-protection officer, a clinician and a lawyer. Her questions revealed that the vendor used customer conversations to improve its models by default and could not provide security test results. The panel required an opt-out, a security testing clause and incident notification within 72 hours before signing; the lawyer assessed AI Act classification separately.

Without a security engineer on the panel, the default training clause might have passed unnoticed.

32.7 Errors and limits

An engineer stating that a system is or is not high-risk under the AI Act oversteps their role and may mislead the organisation. Buying AI tools without security questions leaves risks such as default data reuse undiscovered until after signing. Reports listing problems without options leave managers unable to act, so they often defer decisions. Rina's reports always end with options, their costs and a recommendation, so a manager can approve, reject or ask for changes in one meeting.

Exercises

Synthesis

AI security works through other disciplines: law, privacy, procurement, domain experts and management. Engineers contribute evidence and options; others contribute judgement and authority.

Continue studying — chapter 32 of 40

← Chapter 31 (draft) · Course chapter index · Chapter 33 (draft) →