Chapter 13 — Cryptography and trust services: troubleshooting

Lesson map

Lesson 13 published. Map, example, EX049.

flowchart TD
  N0["Learning outcomes"]
  N1["13.1 The troubleshooting loop"]
  N2["13.2 Triage from logs"]
  N3["Lab — TLS error triage"]
  N4["13.3 Escalation notes"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX049"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["13.1 The troubleshooting loop"]
  N2["13.2 Triage from logs"]
  N3["Lab — TLS error triage"]
  N4["13.3 Escalation notes"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX049"]

[Learning outcomes]
    |
    v
[13.1 The troubleshooting loop]
    |
    v
[13.2 Triage from logs]
    |
    v
[Lab — TLS error triage]
    |
    v
[13.3 Escalation notes]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX049]

Learning outcomes

You will (1) follow a hypothesis-driven troubleshooting loop; (2) triage TLS error logs automatically; (3) escalate with a clear handover note.

13.1 The troubleshooting loop

Priya teaches a loop: observe the symptom, form one hypothesis, run one test that could prove it wrong, record the result and repeat. When depot handhelds suddenly fail to join Wi-Fi, Tomás's first hypothesis is an expired client certificate. He checks one device and finds a valid certificate, so the hypothesis is wrong. His second hypothesis is the RADIUS server's certificate, which the handhelds verify against Lumen's internal root, and it turns out to have been renewed under a new intermediate the devices do not know. Changing one thing at a time made the cause visible quickly.

13.2 Triage from logs

When many failures arrive at once, counting them by reason helps. Tomás writes a small script that reads synthetic load-balancer log lines, extracts the TLS alert or error reason and the client group, and prints counts. Patterns appear immediately: most failures share one reason from one client group, which suggests a single cause rather than many. The lab below shows the output for twenty synthetic lines. Counting is not diagnosis, but it tells him which hypothesis to test first and gives managers a quick sense of scale.

Lab — TLS error triage

# Count synthetic TLS failure reasons by client group.
import collections
groups = ["handheld"] * 12 + ["browser"] * 5 + ["partner"] * 3
reasons = ["unknown_ca"] * 12 + ["certificate_expired"] * 2 + ["handshake_failure"] * 3 + ["protocol_version"] * 3
lines = [f"2026-03-02T06:{i:02d}:00Z lb1 tls_fail client={g} reason={r}" for i, (g, r) in enumerate(zip(groups, reasons))]
c = collections.Counter()
for ln in lines:
    kv = dict(p.split("=") for p in ln.split()[3:])
    c[(kv["reason"], kv["client"])] += 1
print("lines:", len(lines))
for (r, g), n in c.most_common():
    print(f"{n:3d}  {r:20} {g}")

Run with python3 lab_triage.py. Verified output:

lines: 20
 12  unknown_ca           handheld
  3  handshake_failure    browser
  3  protocol_version     partner
  2  certificate_expired  browser

13.3 Escalation notes

If Tomás cannot fix a fault within his authority, he escalates with a handover note. It states the symptom and when it began, the affected users and systems, the hypotheses tested and their results with evidence, what he changed, if anything, and what he recommends next. A good note means the senior engineer does not repeat his tests. Priya reminds him to include time zones and exact error strings, and to write plainly rather than guessing, because a confident but wrong guess can send the next person in the wrong direction.

Worked example

Tomás runs the triage lab, sees that most errors are unknown_ca from handhelds, tests the RADIUS chain hypothesis and writes an escalation note recommending pushing the new intermediate.

Synthetic case. Log lines, devices and the RADIUS incident are synthetic.

Errors and limits

Logs may truncate or omit reasons; counts can mislead if one client retries rapidly. Changing several things at once destroys the evidence trail.

Exercises

• EX049 — One test rule. Why change only one thing per test?
  Show marking criteria

  Rubric: 3 = so the result can be attributed to that change and the cause becomes visible;; 2 = attribution;; 1 = speed;; 0 = none.
• EX050 — Triage top reason. In the lab, which reason and client group dominated?
  Show marking criteria

  Rubric: 3 = unknown_ca from handheld clients, 12 of 20 lines;; 2 = reason only;; 1 = vague;; 0 = none.
• EX051 — RADIUS cause. What caused the handheld Wi-Fi failures?
  Show marking criteria

  Rubric: 3 = RADIUS certificate renewed under a new intermediate unknown to devices;; 2 = mentions RADIUS;; 1 = client cert expired;; 0 = none.
• EX052 — Handover contents. Name four items in an escalation note.
  Show marking criteria

  Rubric: 3 = four of symptom and start time, affected systems, hypotheses with results, changes made, recommendation;; 2 = three;; 1 = one;; 0 = none.

Synthesis

Troubleshooting works by testing one hypothesis at a time, using counts to choose where to look and escalating with complete notes.

Continue studying — chapter 13 of 18

← Chapter 12 (draft) · Course chapter index · Chapter 14 (draft) →