Chapter 13 — Troubleshooting
Lesson map
Lesson 13 published. Map, example, EX049.
flowchart TD
N0["Learning outcomes"]
N1["13.1 Tree"]
N2["13.2 Tool failure"]
N3["13.3 Escalation packet"]
N4["13.4 Workarounds"]
N5["13.5 Pair debugging"]
N6["13.6 Customer impact"]
N7["13.7 Known-good baseline"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX049"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["13.1 Tree"]
N2["13.2 Tool failure"]
N3["13.3 Escalation packet"]
N4["13.4 Workarounds"]
N5["13.5 Pair debugging"]
N6["13.6 Customer impact"]
N7["13.7 Known-good baseline"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX049"]
[Learning outcomes]
|
v
[13.1 Tree]
|
v
[13.2 Tool failure]
|
v
[13.3 Escalation packet]
|
v
[13.4 Workarounds]
|
v
[13.5 Pair debugging]
|
v
[13.6 Customer impact]
|
v
[13.7 Known-good baseline]
|
v
[Worked example] --> [Exercise EX049]
Learning outcomes
You will (1) follow a troubleshooting tree for failed checks; (2) separate tool failure from real findings; (3) escalate with evidence; (4) document workarounds safely.
13.1 Tree
Symptom → reproduce → check scope/account/tool → check staging health → escalate.
In practice the tree starts with scope, then tool health, then the environment. Approval status of the listener is the first question on the troubleshooting card, so Sam checks whether port 8080 was ever approved before debugging why lab_ports reported it. Putting scope first prevents false tool hunts: an hour spent fixing a scanner that correctly flagged an unapproved port is an hour taken from the window for nothing. Only after scope is settled does the card move on to tool and environment checks.
13.2 Tool failure
Certificate errors on the tester laptop are not API findings.
A tool failure, such as a crash or empty output, is logged together with the tool's version pin. Sam then switches to the known-good baseline script, last week's version, whose path is written in the notebook rather than held in memory. Improvising new flags in the middle of an incident is forbidden, because an untested flag can change the traffic pattern and break the rate limits agreed on the method card.
13.3 Escalation packet
Symptom, steps, screenshots, time, what you already ruled out.
Sam's ENG-512 escalation packet has four fields: the ticket identifier, the last good hash, the exact stop message that was posted and the owner's contact details. The last good hash tells on-call which evidence file was intact before the problem, so nobody re-collects it blindly. It takes about five minutes to assemble and saves the on-call engineer a long verbal story. Listing what has already been ruled out stops the engineer repeating checks Sam made an hour earlier.
13.4 Workarounds
Temporary tool flags are logged and removed after; they are not silent permanent config.
Workarounds that change scope are forbidden outright. If a workaround would probe a host that is not on the ENG-512 list, it waits for updated ROE text, and an ROE update request is the only legitimate way a workaround may add a host. Clever scope changes made in the middle of a stuck session are not troubleshooting; they are unapproved testing, and they are recorded as such in the after-action review rather than excused because the original tool was failing.
13.5 Pair debugging
Hard issues get a second technician after fifteen minutes.
Pair debugging puts that second technician on the jump-host view, where they can see the same output and cut a rabbit hole that has burned past the time box. Pair sessions are booked on the calendar before the window opens, so the slot exists before anyone gets stuck. Hoping that a colleague happens to wander past is not pairing, and it usually means the stuck engineer keeps digging alone until most of the window has gone.
13.6 Customer impact
If riders may be affected, stop and call the emergency contact before deep debugging.
Customer impact means riders seeing errors on the journey planner. Any hint of rider-facing harm stops all ENG work immediately, before root-cause analysis continues, and the stop uses the same message template as the auth loop drill. Impact outranks curiosity about an unfinished root cause: the cause can be investigated after the emergency contact has confirmed that riders are no longer affected and staging is stable again.
13.7 Known-good baseline
Sam keeps a known-good curl to the staging health endpoint. If that fails, the environment is broken and findings wait.
Alongside the curl, the known-good baseline is last week's pinned script and the ok lines it printed. When a sudden NOT APPROVED storm appears, Sam opens that pinned output and diffs the new run against it before anyone starts a thread blaming the network. If the old script still prints the expected lines, the change is in the target or the scope; if it does not, the environment is the first suspect.
13.8 Timeboxing ego
After fifteen minutes alone, he must ask for help. Heroic silence wastes engagement window.
The time box exists to protect the ENG-512 window from pride. Fifteen minutes stuck on a tool means escalating to the pair or the lead, not trying one more flag in silence. When ego does delay the call, the after-action review names it plainly and records how many minutes were lost, so the pattern is visible to the whole team. Early escalation is rewarded in that review rather than treated as a sign of weakness.
13.9 Field discipline
Sam's troubleshooting notes use the same Fact/Guess labels as casework. "Guess: WAF blocking my IP" becomes a Fact only after the WAF log shows the block. That discipline keeps him from "fixing" the WAF with volume to see what happens — a behaviour that looks like a DoS and violates the ROE.
Field discipline for troubleshooting is tree, baseline, pair, time box, stop on rider impact. Sam's after-action rewards early escalation. Those habits matter more than solving every fault, because each one keeps a stuck session from turning into unapproved or unsafe testing.
13.10 Closing note
When escalation is required, Sam notifies the engagement lead and the product owner together so nobody hears second-hand that staging is unhealthy.
The closing rule is that troubleshooting exists to protect the window and the systems behind it. Finding the planted fault matters less than stopping when fare alerts appear anywhere, so a fare alert ends troubleshooting even while the plant is still unsolved. An unsolved fault can be picked up in the next session, but a test that kept running through a fare alert is a safety failure that no later fix can undo.
Worked example (synthetic NMT)
Sam's API checks fail with TLS errors; staging is fine in the browser on the jump host. He updates his laptop root store, re-runs, and documents the laptop issue separately from findings.
Synthetic case — Northline Municipal Transit (NMT). The transit agency, Sam Park and all figures are invented for teaching.
Errors and limits
Live transit systems and fare equipment are never tested without a separate, specialised authorization. This book uses synthetic data only and grants no authority to test real systems.
Exercises
- EX049 — First steps. Order: escalate, reproduce, check tool. Rubric: 3 = reproduce, check tool, escalate; 2 = mostly; 1 = escalate first always; 0 = none.
- EX050 — Not a finding. Why is a laptop TLS error not an API finding? Rubric: 3 = client-side trust store; staging works elsewhere; 2 = partial; 1 = it is; 0 = none.
- EX051 — Escalation packet. List four items in an escalation packet. Rubric: 3 = four from chapter; 2 = three; 1 = one; 0 = none.
- EX052 — Rider impact. What if troubleshooting might affect riders? Rubric: 3 = stop; emergency contact; 2 = stop; 1 = continue; 0 = none.
Synthesis
Troubleshoot with a tree, prove where the fault lives and escalate with a clean packet.
Continue studying — chapter 13 of 18
← Chapter 12 (draft) · Course chapter index · Chapter 14 (draft) →