Chapter 4 — Advanced risk and ethics
Lesson map
Lesson 04 published. Map, example, EX016.
flowchart TD
N0["Learning outcomes"]
N1["4.1 Whose risk?"]
N2["4.2 The AI Act in practice"]
N3["4.3 Ethics of testing and disclosure"]
N4["4.4 Management systems"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX016"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["4.1 Whose risk?"]
N2["4.2 The AI Act in practice"]
N3["4.3 Ethics of testing and disclosure"]
N4["4.4 Management systems"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX016"]
[Learning outcomes]
|
v
[4.1 Whose risk?]
|
v
[4.2 The AI Act in practice]
|
v
[4.3 Ethics of testing and disclosure]
|
v
[4.4 Management systems]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX016]
Learning outcomes
You will (1) analyse AI-security risk for affected people, not only the organisation; (2) apply EU AI Act concepts of risk tiers, human oversight (Article 14) and robustness and cybersecurity (Article 15); (3) handle ethical conflicts in testing and disclosure; (4) use ISO/IEC 42001 to embed risk work in management; and (5) defend a risk acceptance decision.
4.1 Whose risk?
Organisational risk registers record losses to the organisation. AI failures often fall on others: a customer wrongly refused, a worker whose data leaks, a driver directed to a wrong address. NIST AI 100-1 asks organisations to consider impacts on individuals, groups, organisations and society. A senior specialist should list affected parties for each risk and ask whether they can notice, contest and recover from the harm.
At Corvane, FraudScore's delays fall on small suppliers, not on Corvane. Amira's risk entry for FraudScore names them and gives them a way to appeal.
4.2 The AI Act in practice
Regulation (EU) 2024/1689 classifies AI systems by risk. High-risk systems carry obligations including risk management, data governance, logging, human oversight — Article 14 requires that people overseeing the system can understand its limits, interpret outputs and intervene or stop it — and Article 15, which requires appropriate accuracy, robustness and cybersecurity, naming threats such as data poisoning, model poisoning, adversarial examples and confidentiality attacks. Whether a given system is high-risk depends on its use; establish this with legal counsel rather than assuming.
For Corvane, whether Dispatch Copilot or FraudScore is high-risk is a question for counsel.
4.3 Ethics of testing and disclosure
Testing must be authorised and scoped; synthetic or consented data should be used; tests on production must not harm users. When you find a weakness in a vendor's model, coordinate disclosure: inform the vendor, agree a timeline, avoid publishing working exploits that endanger others. When internal pressure pushes to hide a finding, document it and escalate: suppressing evidence of a risk to others is an ethical failure even if no rule names it.
At Corvane, Amira's report on the vendor's injection filter was shared with the vendor before any wider discussion. If a test could affect real customers or carriers, stop and use an isolated copy instead, as all of Corvane's tests do.
4.4 Management systems
ISO/IEC 42001 asks organisations to define AI policy, roles, risk assessment and treatment, impact assessment, and continual improvement. Its value for security is that risk work becomes recurring and audited rather than a one-off review.
At Corvane, Amira uses the management system to schedule recurring evaluations for high-tier systems and to record risk acceptances, such as the finance director's acceptance for FraudScore. Audit then checks that these records exist and are current, which keeps the work from fading after the first review.
4.5 Worked example (synthetic Corvane case)
FraudScore flags supplier invoices; flagged suppliers wait 30 days for payment. Amira finds that small suppliers with irregular invoicing are flagged four times as often as large ones. Even if fraud losses fall, the harm lands on small businesses who cannot see why they are delayed. Her risk entry lists them as an affected party; treatment adds a reason code, a 5-day human review and an appeal channel. She records the residual risk and asks the finance director to accept it in writing.
4.6 Errors and limits
- Registers that omit affected people.
- Assuming AI Act classification without legal advice.
- Testing on production users without consent.
Each error leaves harm unmanaged. A register without affected people would miss FraudScore's small suppliers, who bear the delay. Assuming AI Act classification can create obligations that do not apply or miss ones that do; ask counsel. Testing on production users without consent can harm them and breaches the authorisation every test needs. Check your register and test plans for all three.
Exercises
- EX016 — Affected parties. List affected parties and harms for Dispatch Copilot refunds. Rubric: 3 = at least four parties with harms and recourse; 2 = three; 1 = one or two; 0 = none.
- EX017 — Article 14. Describe oversight features for FraudScore that would meet the intent of Article 14. Rubric: 3 = understand limits, interpret, override, stop; 2 = three; 1 = one or two; 0 = none.
- EX018 — Disclosure. You find a jailbreak in a vendor model. Outline a disclosure plan. Rubric: 3 = vendor contact, timeline, limited detail, internal mitigation; 2 = three; 1 = one; 0 = publish immediately.
- EX019 — Pressure. A manager asks you to remove a finding from a report. Respond. Rubric: 3 = documented, escalated, explains harm; 2 = partial; 1 = complies with note; 0 = complies silently.
- EX020 — Acceptance. Write a risk acceptance for the FraudScore residual risk. Rubric: 3 = risk, affected parties, treatment, owner, review date; 2 = four; 1 = two; 0 = none.
Synthesis
Advanced risk work counts harms to everyone affected, uses law and standards as structure rather than decoration, and treats honest testing and disclosure as part of the job.
Continue studying — chapter 04 of 12
← Chapter 03 (draft) · Course chapter index · Chapter 05 (draft) →