◇ OpenCyber Academy
Doctoral research · Chapter 6 · English

Chapter 6 — Peer critique and revision

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 6 — Peer critique and revision

Lesson map

Lesson 06 published. Map, example, EX033.

flowchart TD
  N0["Learning outcomes"]
  N1["6.1 Inviting hostile reading"]
  N2["6.2 The cryptographer's objection"]
  N3["6.3 The methodologist's objection"]
  N4["6.4 The engineer's objection"]
  N5["6.5 Writing the response letter"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX033"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["6.1 Inviting hostile reading"]
  N2["6.2 The cryptographer's objection"]
  N3["6.3 The methodologist's objection"]
  N4["6.4 The engineer's objection"]
  N5["6.5 Writing the response letter"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX033"]

[Learning outcomes]
    |
    v
[6.1 Inviting hostile reading]
    |
    v
[6.2 The cryptographer's objection]
    |
    v
[6.3 The methodologist's objection]
    |
    v
[6.4 The engineer's objection]
    |
    v
[6.5 Writing the response letter]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX033]

Learning outcomes

You will (1) organise structured peer critique of a cryptographic field study; (2) separate fatal objections from fixable ones; (3) revise a manuscript with a traceable response letter.

6.1 Inviting hostile reading

Nadia circulates her draft paper to three readers chosen to disagree with her: an applied cryptographer, a software-engineering methodologist and a Meridian engineer from a control team. She asks each for the single objection most likely to sink the paper and for a list of smaller problems. She also gives them the registration, so they can check whether reported analyses match the plan. Inviting critique before submission is cheaper than receiving it from journal reviewers, and it trains the habit examiners look for: treating one's own result as a claim to be tested.

6.2 The cryptographer's objection

The cryptographer argues that "misuse per ten changes" weights every finding equally, although a repeated GCM nonce on a high-volume key is catastrophic while a deprecated hash in a test utility is cosmetic. Nadia accepts that this is a real construct problem but not a fatal one. She adds a severity-weighted sensitivity analysis using a three-level scale defined in advance of rescoring, and reports that the direction holds, while the gap narrows because several avoided findings were low severity. She also adds a paragraph explaining why the primary, unweighted measure was registered first.

6.3 The methodologist's objection

The methodologist notes that checklist teams merged slightly fewer crypto-touching changes during the quarter, raising the possibility that the checklist slowed work and that slower work, not better review, lowered misuse. Nadia checks merge counts and lead times. Checklist teams merged eight percent fewer such changes, and median review time rose by about twelve minutes. She reports both openly as costs, notes that a rate per ten changes already adjusts for volume, and frames the trade-off for practitioners: a modest time cost bought a sizeable reduction in confirmed misuse within this setting.

6.4 The engineer's objection

The control-team engineer points out that two usual-review teams own the oldest storage service, which carries legacy encryption code with more hazards. Stratification by product line placed both in the control arm by chance. Nadia treats this as a plausible imbalance. She runs a sensitivity analysis excluding the two legacy-heavy teams from control and their stratum partners from treatment; the difference shrinks to about 0.9 but remains positive. She reports the analysis as exploratory, because it was not registered, and recommends that future studies stratify by code age as well as product line.

6.5 Writing the response letter

For each objection Nadia's response letter quotes the comment, states whether she agrees, describes the change with section and line references, and labels any new analysis as exploratory. She resists two temptations: silently promoting a sensitivity result to the abstract, and dismissing an objection without evidence. Where she disagrees, such as a suggestion to drop the unweighted measure, she explains that changing the primary outcome after unblinding would breach the registration. The letter becomes part of her archive, so examiners can trace how the argument evolved.

Worked example

Three critics raise severity weighting, slower merging and a legacy imbalance; Nadia answers with a severity sensitivity check, reported time costs and an exploratory exclusion, all logged in a response letter.

Synthetic note. The reviewers and every rescored count in this chapter are invented.

Errors and limits

Sensitivity analyses run after unblinding are exploratory however reassuring they look. A critique you cannot answer belongs in the limitations, not in a footnote.

Exercises

Synthesis

Structured hostile reading, sensitivity checks labelled as exploratory and a traceable response letter make a result stronger by exposing where it bends.

Continue studying — chapter 06 of 08

← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →