Chapter 10 — Engineering ethics and safety
Lesson map
Lesson 10 published. Map, example, EX020.
flowchart TD
N0["Learning outcomes and connection"]
N1["10.1 Duties"]
N2["10.2 Safety gates and stop criteria"]
N3["10.3 Authorisation is not technical capability"]
N4["10.5 Consent and testing"]
N5["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX020"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["10.1 Duties"]
N2["10.2 Safety gates and stop criteria"]
N3["10.3 Authorisation is not technical capability"]
N4["10.5 Consent and testing"]
N5["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX020"]
[Learning outcomes and connection]
|
v
[10.1 Duties]
|
v
[10.2 Safety gates and stop criteria]
|
v
[10.3 Authorisation is not technical capability]
|
v
[10.5 Consent and testing]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX020]
Learning outcomes and connection
After this chapter you can (1) identify the duties an engineer of response systems holds toward users, data subjects and the public; (2) build authorisation, privacy and service safety into designs and tests; (3) define stop criteria and escalation paths before an intervention; and (4) recognise when a technically correct action is not yours to take. It prepares Chapters 25, 27, 33 and 43.
10.1 Duties
Professional codes such as the ACM Code of Ethics and the NSPE Code of Ethics for Engineers place public welfare and safety first and require honesty about limits. In response engineering those duties have concrete forms: least privilege (tools receive only the authority needed); authorisation (no action outside a delegated scope); minimisation (collect what the decision needs, not everything reachable — a principle in the GDPR, Article 5); and harm awareness (containment can harm people who depend on the contained service).
10.2 Safety gates and stop criteria
A safety gate is a check that must pass before an action proceeds; a stop criterion is a pre-agreed observable that halts an action already under way. Both are written before the incident, because during an incident pressure distorts judgement. A gate for host isolation might require: confirmed asset identity, confirmed owner notified, service criticality looked up, and an approver who is not the requester. A stop criterion might be: "if the clinical scheduling queue exceeds 50 waiting requests or any ward reports inability to view schedules, restore connectivity and escalate."
10.3 Authorisation is not technical capability
Your tooling may be able to disable any account. That does not mean you may. Authority comes from policy, contracts and law; engineering's job is to make the authorised path easy and the unauthorised path impossible or loud. Where authority is ambiguous, the design must route the decision to someone who has it. In EX020, the HVSS engineer's tooling may reach the second client's tenant, yet only that client can authorise action there. The right design there routes the observation to that client through the defined notification channel rather than acting.
10.4 Worked example (synthetic): isolating a clinical scheduling service
A synthetic HVSS client hospital runs a care-scheduling service. An alert suggests a compromised service account on its application server. The automatic playbook would isolate the server. Gates: (1) asset criticality = patient-facing → automatic isolation forbidden; (2) clinical service owner and on-call clinician lead must be contacted; (3) alternative: rotate the service account credential and block its outbound internet access, which reduces attacker capability without stopping scheduling; (4) stop criterion as above; (5) escalation path: response manager → hospital executive on call within 15 minutes if the owner cannot be reached. The decision record names who approved what and when. The engineering lesson: the playbook gained a criticality check and a less disruptive alternative action; the safety property "patient-facing services are never isolated without clinical approval" became a test.
10.5 Consent and testing
Tests and exercises involving real people or systems need agreement from those accountable, a defined scope and a way to stop. The Menlo Report adapts research-ethics principles — respect for persons, beneficence, justice and respect for law and public interest — to ICT work, and is a useful checklist for exercises and practicums (Chapter 43).
At HVSS, an exercise on the hospital trust's care-scheduling service would need agreement from the clinical service owner and the response manager. It would also need a scope limited to named test accounts and the same stop criterion of 50 waiting requests used for real isolation decisions.
10.6 Errors and limits
- Equating "the tool allows it" with "I am allowed".
- Stop criteria written after the action started.
- Collecting broadly "in case".
- Ethics treated as a sign-off rather than a design input.
The care-scheduling case shows how each error would have played out. The automatic playbook could isolate the application server, which is exactly the equation of tool capability with permission. Writing the 50-request stop criterion only after isolation had begun would leave wards without schedules while people argued about thresholds. In that case, ethics changed the design itself by adding a criticality check and a less disruptive credential rotation.
Exercises — chapter 10
EX019 — Gates and stops. Objective: design safety into a playbook. Inputs: a synthetic water-billing portal whose web server shows signs of web-shell activity; the portal also handles emergency leak reports. Steps: write three gates, two stop criteria, a less disruptive alternative and an escalation path. Expected evidence: list. Rubric: 3 = specific, observable, with alternative that preserves leak reporting; 2 = generic; 1 = one element; 0 = none.
EX020 — Authority. Objective: recognise limits. Inputs: a managed-service engineer at HVSS sees, while handling a client case, that another client's account is used from the same attacker infrastructure. Steps: state what the engineer may and may not do, and what the system should support. Expected evidence: short argument. Rubric: 3 = no action in the other tenant without its authorisation; notify through the defined channel; system supports cross-tenant indicator sharing without data exposure; 2 = partial; 1 = acts directly; 0 = none.
Chapter synthesis
Ethics in response engineering is concrete: least privilege, authorisation, minimisation and harm awareness, encoded as gates, stop criteria and escalation paths that are tested like any other requirement.
Continue studying — chapter 10 of 50
← Chapter 09 · Course chapter index · Chapter 11 →
Continue to the next authored lesson.