Chapter 20 — Quantum and post-quantum security: troubleshooting | Technologist Quantum and post-quantum security

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 20 published. Map, example, EX039.

flowchart TD
  N0["Learning outcomes"]
  N1["20.1 Method"]
  N2["20.2 Practical activity — Lab H"]
  N3["20.3 Common causes"]
  N4["20.4 Documenting"]
  N5["20.7 Escalating to vendors"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX039"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["20.1 Method"]
  N2["20.2 Practical activity — Lab H"]
  N3["20.3 Common causes"]
  N4["20.4 Documenting"]
  N5["20.7 Escalating to vendors"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX039"]

[Learning outcomes]
    |
    v
[20.1 Method]
    |
    v
[20.2 Practical activity — Lab H]
    |
    v
[20.3 Common causes]
    |
    v
[20.4 Documenting]
    |
    v
[20.7 Escalating to vendors]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX039]

Learning outcomes

You will (1) apply a systematic troubleshooting method to handshake failures; (2) triage failure logs into likely causes; (3) diagnose middlebox, size, group-mismatch and certificate problems; and (4) document fixes and residual issues.

20.1 Method

Reproduce, isolate, hypothesise, test, fix, verify, document. Reproduce in the lab with the same client and path; isolate by removing hops; form a hypothesis tied to evidence; change one thing; verify with negotiation logs; document the cause and fix.

Applied at KLS, the steps stop the reflex fix of disabling hybrid. The proxy-b case was reproduced with a lab replica, isolated by sending the same client directly, and the hypothesis, a dropped second segment, was confirmed before any change. Verification used the negotiation logs, not the absence of complaints.

20.2 Practical activity — Lab H

# Lab H (ch 20): triage synthetic handshake failure logs into likely causes.
import re, collections
logs = """2026-10-01T10:00Z client=10.0.4.7 err=handshake_timeout size_clienthello=1530 path=proxy-b
2026-10-01T10:01Z client=10.0.4.9 err=handshake_timeout size_clienthello=1530 path=proxy-b
2026-10-01T10:02Z client=10.0.7.2 err=no_shared_group offered=x25519mlkem768
2026-10-01T10:03Z client=10.0.4.8 err=handshake_timeout size_clienthello=1530 path=proxy-b
2026-10-01T10:04Z client=10.0.9.1 err=bad_certificate alg=mldsa65
2026-10-01T10:05Z client=10.0.4.3 err=handshake_timeout size_clienthello=300 path=direct""".splitlines()
causes = collections.Counter()
for line in logs:
    f = dict(re.findall(r"(\w+)=(\S+)", line))
    if f["err"] == "handshake_timeout" and int(f.get("size_clienthello", 0)) > 1400: c = "large ClientHello dropped (" + f["path"] + ")"
    elif f["err"] == "no_shared_group": c = "client/server group mismatch"
    elif f["err"] == "bad_certificate": c = "client cannot verify PQ certificate"
    else: c = "other / investigate"
    causes[c] += 1
for c, n in causes.most_common(): print(f"{n}  {c}")

Expected output:

3  large ClientHello dropped (proxy-b)
1  client/server group mismatch
1  client cannot verify PQ certificate
1  other / investigate

Three failures cluster on one proxy with large ClientHellos; that pattern points to a middlebox mishandling multi-segment messages.

The triage rules are ordered from specific to general. A timeout with a ClientHello over 1,400 bytes is attributed to the path, which groups the three proxy-b lines; the identical timeout at 300 bytes on the direct path falls through to "other", because a small hello cannot be explained by size. no_shared_group names the client offering only the hybrid group against a server that does not accept it, and bad_certificate with alg=mldsa65 points at a client that cannot verify the new chain. The value of the script is the grouping: one path explains half the failures.

20.3 Common causes

Large ClientHello dropped by old firewalls or proxies: upgrade, bypass or adjust. No shared group: client and server lists do not overlap (e.g. a server configured hybrid-only and a classical-only client). Certificate verification failure: client cannot verify ML-DSA or lacks the new root. HelloRetryRequest loops or latency: client key-share prediction mismatches. Performance: CPU spikes on constrained devices.

For each cause KLS records the evidence that distinguishes it: segment-size thresholds and path for drops, the two group lists for mismatches, the certificate algorithm and client trust store for verification failures, timing with a HelloRetryRequest for latency, and CPU graphs for performance.

20.4 Documenting

Each incident record states symptoms, evidence, cause, fix, verification and residual issues. Patterns across incidents inform vendor requests and rollout plans.

KLS keeps incident records in the same repository as runbooks. After three proxy cases from the same appliance model, the pattern record became a vendor request and a pre-rollout check: any site behind that model gets a lab test before the profile reaches it.

Each record links to its lab reproduction and to the runbook it changed. A ticket closed without verification evidence is reopened by the weekly review, because "no new complaints" is not evidence that negotiations recovered.

20.5 Worked example

KLS ticket: "Warehouse office cannot reach portal since Tuesday." Evidence: timeouts only through proxy-b; ClientHello 1,530 bytes; direct path works. Hypothesis: proxy-b drops second segment. Test: lab replica of proxy-b firmware reproduces. Fix: proxy firmware update; interim: portal bypass rule for that office. Verified: hybrid negotiations from office resume. Residual: two other offices with the same proxy model scheduled for update.

The interim bypass was scoped to one office's address range and given an expiry, so it could not quietly become permanent. The verification step checked both directions: hybrid negotiations from the office appeared in the portal's logs, and the failure count for proxy-b fell to zero over the following week.

20.6 Common errors and limits

Disabling hybrid globally to fix one path.
Changing several things at once.
Fixes without verification.

During the proxy-b ticket someone proposed removing X25519MLKEM768 from the portal profile; that would have fixed one office and re-exposed every other client to harvest-now-decrypt-later. A second engineer upgraded the proxy and changed the portal profile in the same window, making the cause impossible to attribute. And an earlier ticket had been closed as fixed after the complaints stopped, though logs showed the office simply had a holiday.

20.7 Escalating to vendors

When the cause lies in a product you do not control, escalate with evidence: product and firmware version, a minimal reproduction in the lab, captures from lab traffic only, and the expected behaviour with a reference to the relevant standard (for example RFC 8446 on ClientHello handling, or RFC 10024 for the hybrid group). Precise reports get fixed faster, and they become part of the residual-risk record until the fix ships.

KLS's report on proxy-b's model gave firmware version, the 1,530-byte lab ClientHello and RFC 8446's requirement to handle multi-record handshake messages.

Exercises

EX039 — Diagnose. A partner reports no_shared_group after KLS changed SFTP settings. Give the likely cause and fix. Rubric: 3 = hybrid-only or removed classical, restore fallback or upgrade partner.
EX040 — Extend Lab H. Add a rule classifying err=hrr_timeout as "HelloRetryRequest latency". Rubric: 3 = correct code and output.

Synthesis

Systematic troubleshooting, log triage and documented fixes resolve post-quantum handshake failures without abandoning protection globally.

Continue studying — chapter 20 of 30

← Chapter 19 (draft) · Course chapter index · Chapter 21 (draft) →