Chapter 10 — Engineering ethics and safety
Lesson map
Lesson 10 published. Map, example, EX028.
flowchart TD
N0["Learning outcomes"]
N1["10.1 Professional duties"]
N2["10.2 Safety and security"]
N3["10.3 Authorised testing and disclosure"]
N4["10.4 Law and regulation"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX028"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["10.1 Professional duties"]
N2["10.2 Safety and security"]
N3["10.3 Authorised testing and disclosure"]
N4["10.4 Law and regulation"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX028"]
[Learning outcomes]
|
v
[10.1 Professional duties]
|
v
[10.2 Safety and security]
|
v
[10.3 Authorised testing and disclosure]
|
v
[10.4 Law and regulation]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX028]
Learning outcomes
You will (1) apply professional engineering ethics to AI systems; (2) distinguish safety (accidental harm) from security (intentional harm) and show how they interact; (3) apply rules for authorised testing and responsible disclosure; and (4) recognise legal duties such as those in the EU AI Act for high-risk systems.
10.1 Professional duties
Engineering codes of ethics put public safety, health and welfare first, require honesty about capabilities and limits, and require competence. For AI this means not overstating what a model can do, documenting known failure modes, and refusing to deploy where risk is unacceptable.
These duties apply in ordinary moments, not only dramatic ones: a slide that rounds a 9% dawn miss rate down to 'about 2%', a demonstration that hides the conditions where the model fails, or a deadline that tempts a team to skip the safety review. Competence includes knowing the limits of one's own expertise and asking for help, for example from a rail safety engineer, when a decision falls outside it.
10.2 Safety and security
Safety engineering addresses accidental failure; security addresses deliberate attack. In AI they interact: an adversarial input can cause a safety failure (a missed rail defect), and a safety mechanism (a fallback to manual inspection) can be abused to cause denial of service. The NIST AI RMF (AI 100-1) lists safe and secure and resilient among trustworthy-AI characteristics and stresses managing them together.
Managing them together means one hazard analysis that asks both what could fail and what someone could make fail. A fallback to manual inspection is safe against accidents, but an attacker who can trigger it repeatedly could exhaust the inspectors and delay real work.
10.3 Authorised testing and disclosure
Test only systems you own or have written permission to test, within a defined scope and time. Never run attacks against public AI services without authorisation; terms of service often forbid it. When you find a vulnerability in someone else's product, report it privately to the vendor and agree a timeline before publishing (coordinated disclosure). Keep test data synthetic or properly governed; never extract or publish personal data.
Written permission names who may test, what, when and by which methods.
10.4 Law and regulation
Regulation (EU) 2024/1689 (the AI Act) classifies some AI systems as high-risk — including safety components of critical infrastructure — and requires risk management, data governance, technical documentation, logging, human oversight, and accuracy, robustness and cybersecurity (Article 15). Obligations apply in stages from 2025 to 2027. Engineers should know whether their system may fall into a regulated category and involve legal experts; this book gives orientation, not legal advice.
Even before legal advice, an engineer can prepare the evidence a regulated system would need: a risk register, dataset documentation, logs of decisions and records of human review. Those records are good practice anyway, and gathering them early avoids rebuilding a system's history later under deadline.
10.5 Worked example (synthetic NRS case)
A manager asked Sam to raise TrackEye's alert threshold to reduce false alarms before a demonstration. Sam computed that false negatives would rise from 2% to 5% on daytime data. Sam documented the trade-off, declined to change the production threshold without a safety review, and proposed a demonstration-only configuration clearly labelled as such. Sam also noted that TrackEye, as a safety component of rail infrastructure, may be high-risk under the AI Act, and flagged this for legal review.
The demonstration configuration resolved the manager's goal without touching the system that inspects real track.
10.6 Errors and limits
- Treating ethics as separate from technical decisions.
- Testing third-party AI services without permission.
- Hiding safety trade-offs in configuration changes.
Ethics separated from technical work tends to appear only in policy documents, while the actual decisions are made in thresholds and test plans. Testing a public AI service without permission can break the law even with good intentions. A threshold change that silently trades missed defects for fewer alarms is a safety decision made without the people responsible for safety; Sam's rule is that any change affecting miss rates goes through review with its numbers stated.
Exercises
- EX028 — Safety vs security. Give one example where a security attack causes a safety failure at NRS and one where a safety mechanism is abused. Rubric: 3 = both plausible; 2 = one; 1 = confused; 0 = none.
- EX029 — Disclosure. You find that a vendor's hosted model leaks other customers' prompts. List the steps you take. Rubric: 3 = stop, preserve minimal evidence, private report, agreed timeline, no publication of data; 2 = partial; 1 = vague; 0 = publish or exploit.
- EX030 — AI Act. Name four obligations for high-risk AI systems. Rubric: 3 = four of risk management, data governance, documentation, logging, transparency, human oversight, accuracy/robustness/cybersecurity; 2 = three; 1 = one or two; 0 = none.
Synthesis
Ethics in AI engineering is concrete: honest limits, safety and security managed together, authorised testing, coordinated disclosure, and awareness of legal duties for high-risk systems.
Continue studying — chapter 10 of 50
← Chapter 09 (draft) · Course chapter index · Chapter 11 (draft) →