Chapter 6 — Peer critique and revision
Lesson map
Lesson 06 published. Map, example, EX033.
flowchart TD
N0["Learning outcomes and connection"]
N1["6.1 Objections as tests"]
N2["6.2 Red-team your own claims"]
N3["6.3 The architectural boundary"]
N4["6.4 Narrowing a claim"]
N5["6.5 Dual-use review"]
N6["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX033"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["6.1 Objections as tests"]
N2["6.2 Red-team your own claims"]
N3["6.3 The architectural boundary"]
N4["6.4 Narrowing a claim"]
N5["6.5 Dual-use review"]
N6["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX033"]
[Learning outcomes and connection]
|
v
[6.1 Objections as tests]
|
v
[6.2 Red-team your own claims]
|
v
[6.3 The architectural boundary]
|
v
[6.4 Narrowing a claim]
|
v
[6.5 Dual-use review]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX033]
Learning outcomes and connection
After this chapter you can (1) convert reviewer objections into tests; (2) run an internal red-team review of your own claims; (3) find the architectural boundary beyond which a method cannot see; (4) narrow a claim without abandoning it; and (5) conduct a dual-use review of an inventory method. It takes the results of Chapter 5 and prepares the thesis argument of Chapter 7.
6.1 Objections as tests
A reviewer's objection is a hypothesis about your error. Treat each one as a test with a possible outcome, not as an attack to rebut. Three objections from the synthetic review of the Ciphertrail paper illustrate the method. O1: "Passive telemetry may find appliances only because HRHN's appliances happen to sit on the monitored segment." Test: re-run the analysis restricted to appliances on segments with and without sensors; the gain vanishes off-sensor, confirming that the method's benefit is conditional on sensor placement — a scope condition, now stated. O2: "Adjudicators could infer provenance from formatting." Test: give 50 items to two new engineers with formatting normalized; κ with the original labels is 0.81, so the objection is weakened but recorded. O3: "Recall of uses is not risk; one RSA-wrapped archive matters more than a hundred internal web servers." This is not testable within the data; it is a construct objection, answered by adding a sensitivity-weighted recall and conceding that weighting is a decision, not a measurement.
6.2 Red-team your own claims
Before external review, assign a colleague to argue that each claim is false. A useful red-team table has four columns: claim, strongest objection, test or concession, status after revision. Red-teaming in PQC research should always include an objection from the threat-model side: hybrid negotiation protects key establishment only if the classical and post-quantum shares are combined as specified and no downgrade path exists. An inventory that records X25519MLKEM768 as negotiated says nothing about whether a client would silently accept a classical-only group from an active attacker — a separate property that inventory does not measure.
6.3 The architectural boundary
Every method has a place it cannot see. For dual-source inventory the boundary includes: TLS 1.3 certificate signatures unless endpoints cooperate; firmware-update signatures verified offline inside devices; data at rest encrypted by third-party services; and protocols carried inside VPN tunnels the sensor cannot open. State the boundary as a diagram and a list in the thesis. Examiners respect a candidate who shows where the method stops more than one who implies it covers everything.
Draw the diagram with each blind zone labelled by the observer that misses it.
6.4 Narrowing a claim
Original claim: "Dual-source inventory substantially improves discovery of quantum-vulnerable cryptography." Narrowed claim after critique: "Where passive sensors observe the relevant segments, dual-source inventory increased recall of quantum-vulnerable key-establishment and TLS-1.2-visible signature uses by 0.10–0.21 in two synthetic organizations, concentrated in vendor appliances; it did not improve discovery of data-at-rest or offline firmware-signature uses." The narrowed claim is smaller and much harder to refute — exactly the trade a thesis should make.
Check the title, abstract and conclusions against the narrowed claim before submission.
6.5 Dual-use review
An inventory method is a map of where an organization is weakest. Published inventories, or tooling that fingerprints legacy cryptography across the internet, could guide a harvest-now-decrypt-later collector to the highest-value targets. The dual-use review asks: does the release include organization-identifying data (it must not); does the tool's default configuration scan only owner-authorized targets; are weaknesses in named products reported to vendors through coordinated disclosure (ISO/IEC 29147 describes vulnerability disclosure) before publication; and does the paper describe classes of weakness rather than lists of vulnerable hosts?
6.6 Worked example: the response letter
The synthetic response letter opens with a table mapping each objection to a change: O1 → new scope condition and an off-sensor analysis (Section 5.4 of the paper); O2 → new robustness check, κ reported; O3 → added sensitivity-weighted recall, and the abstract now says "uses" rather than "risk". It thanks reviewers without flattery, concedes what was right, and explains disagreements with evidence. Where the authors disagree — a reviewer asked for active scanning of medical devices to "complete" the ground truth — the letter explains the harm argument from Chapter 1 and offers a testbed experiment instead.
6.7 Common errors and limits
- Rebutting objections rhetorically instead of testing them.
- Narrowing a claim in the discussion but not in the title and abstract.
- Confusing inventory of negotiated algorithms with resistance to downgrade.
- Publishing per-organization findings that become a targeting list.
- Red-teaming only by friends who share your assumptions.
Each error has a remedy already used in the Ciphertrail review: objections O1 and O2 became tests, the narrowed claim moved into the abstract, downgrade resistance was named as outside the method, and the dual-use review removed organization-identifying data. For red-teaming, ask someone from the threat-model side who does not share your assumptions.
Exercises — chapter 6
- EX033 — Objection to test. Convert three objections of your own into tests with possible outcomes. Rubric: 3 = each test could change the conclusion.
- EX034 — Red-team table. Build a four-column red-team table for the three main Ciphertrail claims. Rubric: 3 = includes a threat-model objection.
- EX035 — Boundary diagram. Draw (or describe in text) the architectural boundary of dual-source inventory with at least five unseen zones. Rubric: 3 = zones accurate and each linked to a scope statement.
- EX036 — Narrow it. Narrow "Hybrid TLS makes our hospital quantum-safe" into a defensible claim. Rubric: 3 = names the property, scope and what remains vulnerable.
- EX037 — Dual-use review. Complete a dual-use review for releasing the HRHN corpus. Rubric: 3 = decides what is released and why.
- EX038 — Response letter. Write the letter section answering O3. Rubric: 3 = concedes the construct point and shows the change.
Chapter synthesis
Critique improves a thesis when objections become tests, the method's blind zone is drawn explicitly, claims shrink to what survives, and the release is reviewed for harm. Inventory evidence is not evidence of downgrade resistance or of risk reduction without further argument.
Continue studying — chapter 06 of 08
← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →