Chapter 10 — Ethics and authorized practice
Lesson map
Lesson 10 published. Map, example, EX028.
flowchart TD
N0["Learning outcomes"]
N1["10.1 Principles"]
N2["10.2 Authorization"]
N3["10.3 Coordinated disclosure"]
N4["10.4 Fairness and safety"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX028"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["10.1 Principles"]
N2["10.2 Authorization"]
N3["10.3 Coordinated disclosure"]
N4["10.4 Fairness and safety"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX028"]
[Learning outcomes]
|
v
[10.1 Principles]
|
v
[10.2 Authorization]
|
v
[10.3 Coordinated disclosure]
|
v
[10.4 Fairness and safety]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX028]
Learning outcomes
You will (1) apply ethical principles to AI-security work; (2) explain why authorization is required before testing; (3) follow coordinated disclosure; and (4) recognise fairness, privacy and safety issues raised by AI systems.
10.1 Principles
Respect for people, avoiding harm, honesty, fairness and accountability guide security work. In AI security, harms include privacy breaches, discriminatory outcomes, unsafe advice and misuse of capabilities.
These principles turn into concrete choices. Avoiding harm means testing on synthetic records, not real patients. Honesty means reporting results with their limits. Fairness means checking whether NoShow's predictions treat groups differently. Accountability means recording who authorised each test and who made each decision, so responsibility is clear if something goes wrong. Rina's lab notes record each of these choices.
10.2 Authorization
Testing systems without permission can be illegal and can harm real users. Always obtain written authorization from the system owner that defines scope, methods, time and contacts. Public AI services have terms of use; do not attack them outside programmes that explicitly permit it (such as published bug-bounty scopes).
Rina's authorisation for LCH work names the systems (isolated copies of Ask Lumen, NoShow and DocSort), the methods allowed, the dates and the contact to call if something unexpected happens. She carries a copy during every test session.
10.3 Coordinated disclosure
If you find a vulnerability, report it privately to the owner, give reasonable time to fix, and publish only details that help defenders. Do not access more data than needed to demonstrate the problem.
In Rina's case, the phone-number issue was demonstrated with a single synthetic example rather than by checking many real numbers. Her report described the behaviour, the conditions and a suggested fix, and she kept it confidential until LCH confirmed the fix, which came before the semester ended.
10.4 Fairness and safety
AI systems can treat groups differently. NoShow, for example, could rank some patients lower for reminders because of features correlated with income. The NIST AI RMF lists fair, with harmful bias managed among trustworthy-AI characteristics alongside secure and resilient and privacy-enhanced. The EU AI Act imposes obligations on high-risk systems, including some healthcare-related uses; legal classification needs expert advice.
A fairness check for NoShow would compare error rates across groups, for example whether patients from some postcodes are more often wrongly predicted to miss appointments.
10.5 Worked example (synthetic LCH case)
A classmate proposed testing a commercial health chatbot "to see if it leaks data". Rina explained that without authorization this would be unethical and possibly illegal, and suggested building a synthetic replica in the course lab instead. Separately, in an authorised LCH test, Rina found that Ask Lumen could reveal whether a phone number belonged to a registered patient; she reported it through LCH's process with minimal evidence, and it was fixed before the semester ended.
Minimal evidence meant one synthetic phone number, not a list of real ones.
10.6 Errors and limits
- "Just checking" unauthorised systems.
- Collecting more personal data than needed for a report.
- Ignoring fairness as "not security".
"Just checking" an unauthorised system can break the law and harm users, regardless of intent. Collecting more personal data than needed turns a security report into a privacy risk of its own. Ignoring fairness overlooks harms that affect patients as directly as a data leak would, such as NoShow giving some groups fewer reminders because of features correlated with income. Rina's practice addresses all three by design.
Exercises
- EX028 — Authorization. List five elements of a testing authorization. Rubric: 3 = scope, methods, time window, data handling, contacts/stop conditions; 2 = four; 1 = two; 0 = none.
- EX029 — Disclosure. Describe Rina's disclosure steps for the phone-number issue. Rubric: 3 = private report, minimal evidence, fix verification, no public data; 2 = partial; 1 = vague; 0 = public disclosure first.
- EX030 — Fairness. Explain how NoShow could create unfair outcomes. Rubric: 3 = proxies for protected characteristics affect reminders/scheduling; 2 = partial; 1 = vague; 0 = none.
Synthesis
Ethical AI-security practice means authorised testing, careful disclosure, minimal data and attention to fairness and safety as well as attacks.
Continue studying — chapter 10 of 40
← Chapter 09 (draft) · Course chapter index · Chapter 11 (draft) →