Chapter 8 — Failure modes | Technical Authorized offensive security and red team

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 8 — Failure modes

Lesson map

Lesson 08 published. Map, example, EX029.

flowchart TD
  N0["Learning outcomes"]
  N1["8.1 Harm modes"]
  N2["8.2 Stop conditions"]
  N3["8.3 Safe tool failure"]
  N4["8.4 Communicating stop"]
  N5["8.5 Rehearsal"]
  N6["8.6 No blame in the moment"]
  N7["8.7 Secondary contact"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX029"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["8.1 Harm modes"]
  N2["8.2 Stop conditions"]
  N3["8.3 Safe tool failure"]
  N4["8.4 Communicating stop"]
  N5["8.5 Rehearsal"]
  N6["8.6 No blame in the moment"]
  N7["8.7 Secondary contact"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX029"]

[Learning outcomes]
    |
    v
[8.1 Harm modes]
    |
    v
[8.2 Stop conditions]
    |
    v
[8.3 Safe tool failure]
    |
    v
[8.4 Communicating stop]
    |
    v
[8.5 Rehearsal]
    |
    v
[8.6 No blame in the moment]
    |
    v
[8.7 Secondary contact]
    |
    v
[Worked example] --> [Exercise EX029]

Learning outcomes

You will (1) list how tests can harm transit IT; (2) map stop conditions; (3) describe safe failure of tools; (4) practise communicating a stop.

8.1 Harm modes

Tests can harm transit IT through account lockouts, staging crashes, an accidental production hit, log storms and poisoned caches. For ENG-512 the harm modes include API lockout, VPN saturation and any touch of fare validators. Each of Sam's method cards lists which harm its check could cause, so stop conditions are not invented under stress. Cards without harm modes are incomplete for ENG-512 and stay off the Saturday list until the author adds them. A card's harm line also names who must be told first if that harm occurs, for example the API owner for a lockout.

8.2 Stop conditions

Stop conditions are: the owner asks to stop; an unexplained outage; fare system alerts anywhere; personal data exposed in a ticket; a tool loop. Each condition has a one-line stop message prepared on the ROE card, so Sam posts it without composing under stress. Any one is enough; Sam does not finish one more objective first. Fare system alerts anywhere, even apparently unrelated ones, stop ENG-512 until operations clears the line, and Sam does not argue correlation in the moment. Personal data turning up in a ticket is a stop even though nothing is broken, because the ticket system may be read by people the agreement never covered. A tool caught in a loop is a stop for a different reason: it can push traffic past the agreed rate before anyone notices.

8.3 Safe tool failure

Prefer tools that fail closed (stop on error) rather than retrying into a storm.

Safe tool failure means a script exits and releases its hosts when errors repeat, rather than retrying forever against staging. The 401 loop drill teaches exactly this: when the auth checks loop, Sam's script drops its sockets and prints a halt reason that names the cause. A tool that keeps retrying into a storm is treated as defective, however thorough it looks, because the extra traffic eats the noise budget and hides the original error from the responders who have to read the logs.

8.4 Communicating stop

Say what you were doing, what you observed, what you halted, and ask for guidance — short and factual.

The stop message goes to the engagement channel in a fixed shape: ENG-512 stop, cause, halted HH:MM, need guidance. On the drill it reads "ENG-512 stop — looping auth checks on staging API — halted 10:12 — need guidance." Softening words and apologies are deleted before sending, because the on-call responder has to act on the first line. A factual header lets them see what stopped, when it stopped and what decision is now waiting on them.

8.5 Rehearsal

Once per quarter Sam rehearses a stop message on the engagement channel with the SOC, and the stop message is also rehearsed in Lab II before a real window. Sam practises posting the 10:12 pattern so the first live stop is not the first time he writes one, and Lab II grades the rehearsal stop post before any live window. An unrehearsed first stop is a programme failure waiting to happen. The SOC rehearses its side too: acknowledging the post and naming who decides.

8.6 No blame in the moment

Stop first; investigate cause after systems are stable.

No blame belongs in the moment. During ENG-512 the live channel carries only halt facts and the request for guidance, and any blame language that appears in the stop thread is moderated out. Discussion of why the auth checks looped waits for the after-action review, which owns the learning. Separating the two keeps responders focused on stabilising staging, and it gives the later review a clean timeline instead of an argument recorded in the middle of the incident.

8.7 Secondary contact

If the primary emergency contact does not answer in five minutes, Sam calls the secondary named in the ROE; waiting silently is not a plan. The five-minute mark is timed from the first unanswered call, and both call times go into the activity log. Sam's ROE lists both contacts, and calling only the primary is not a complete stop path. He dials both numbers from the ROE card, not from memory. If neither answers, the halt stays in place and Sam keeps calling; the window does not resume by default.

8.8 Tool kill switch

Every long-running script Sam starts has a known way to kill it, such as a process group or a container stop, and he tests the kill switch once before the window. For ENG-512 the kill switch is the script's disable flag plus the jump-host kill command practised in rehearsal, so Sam can stop his own jobs without waiting for a vendor support line. Kill-switch rehearsal is a Lab II checkbox, and an untested disable flag keeps automation off the Saturday list.

8.9 Field discipline

After any real stop, Sam files a short blameless note: trigger, time to halt, who was called, whether riders were affected, and one process change. NMT reviews stop notes quarterly. Teams that hide stops eventually cause larger outages; teams that record them get better kill switches and clearer ROEs. The looping-auth drill exists so the first stop is not also the first time Sam writes that note. Field discipline for failure modes is written stop conditions, rehearsed messages, a secondary contact and a kill switch, each checked before the Saturday window opens, so ENG-512 does not rely on heroics or improvisation when a loop starts.

Worked example (synthetic NMT)

During a synthetic drill, Sam's script loops on 401s. He stops the script, posts "ENG-512 stop — looping auth checks on staging API — halted 10:12 — need guidance", and waits.

Synthetic case — Northline Municipal Transit (NMT). The transit agency, Sam Park and all figures are invented for teaching.

Errors and limits

Live transit systems and fare equipment are never tested without a separate, specialised authorization. This book uses synthetic data only and grants no authority to test real systems.

Exercises

EX029 — Harm list. Name three test harm modes. Rubric: 3 = three from chapter; 2 = two; 1 = one; 0 = none.
EX030 — Fare alert. A fare alert appears during a web test. What does Sam do? Rubric: 3 = stop; contact emergency; do not touch fare; 2 = stop; 1 = continue web; 0 = none.
EX031 — Stop message. Which four facts go in a stop message? Rubric: 3 = action, observation, halted, ask; 2 = three; 1 = one; 0 = none.
EX032 — Fail closed. What does fail closed mean for a test tool? Rubric: 3 = stops on error instead of aggressive retry; 2 = partial; 1 = keeps going; 0 = none.

Synthesis

Stops are rehearsed, fast and factual; tools should fail closed.

Continue studying — chapter 08 of 18

← Chapter 07 (draft) · Course chapter index · Chapter 09 (draft) →