Chapter 6 — Peer critique and revision

Lesson map

Lesson 06 published. Map, example, EX033.

flowchart TD
  N0["Learning outcomes"]
  N1["6.1 The critique"]
  N2["6.2 Power"]
  N3["6.3 Comparator and mechanism"]
  N4["6.4 Revised claims"]
  N0 --> N1 --> N2 --> N3 --> N4
  N4 --> EX["Worked example"]
  EX --> Q["Exercise EX033"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["6.1 The critique"]
  N2["6.2 Power"]
  N3["6.3 Comparator and mechanism"]
  N4["6.4 Revised claims"]
  N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX033"]

[Learning outcomes]
    |
    v
[6.1 The critique]
    |
    v
[6.2 Power]
    |
    v
[6.3 Comparator and mechanism]
    |
    v
[6.4 Revised claims]
    |
    v
[Worked example] --> [Exercise EX033]

Learning outcomes

You will (1) respond to critical peer review; (2) revise analysis and claims without changing confirmatory results; (3) separate exploratory findings from confirmatory ones.

6.1 The critique

Yusuf presents a draft paper at a departmental seminar and submits it to a workshop. Reviewers raise three points. First, the study may be underpowered: with this much variation, a three-day effect would often be missed. Second, the comparison departments include the water utility, whose patching is constrained by safety testing, making it a poor comparator. Third, the paper does not explain what risk-ranked patching changed in practice, so readers cannot tell what was actually tested. Yusuf logs each point with a planned response before revising.

6.2 Power

Yusuf runs a simulation, generating synthetic data with a true three-day effect and the observed spread, and counts how often his design detects it. Detection is well below the conventional eighty percent target. He adds this to the paper, stating that the study could not reliably detect effects of the size observed. He recommends that future studies pool several organisations. Prof. Hald notes that reporting power after an inconclusive result helps readers avoid concluding that the practice has no effect. In his simulation, detection occurs about thirty percent of the time.

6.3 Comparator and mechanism

On the comparator, Yusuf runs a sensitivity analysis excluding the water utility, labelled clearly as exploratory because it was not pre-registered. The estimate changes slightly but the interval still includes zero. On mechanism, he adds a section describing what changed: vulnerabilities were scored against each system's FIPS-199-style category, high systems were patched first and weekly reports went to department heads. He interviews two security officers to document practice, marking interview material as descriptive rather than causal evidence. These additions are marked as revisions.

6.4 Revised claims

The revised paper keeps the pre-registered result unchanged as the main finding, adds the power simulation, the exploratory sensitivity analysis and the mechanism description, and narrows the conclusion: in one council, risk-ranked patching was associated with a modest, uncertain reduction in critical patch latency. The workshop accepts the paper. Yusuf notes that critique improved the paper mainly by making its limits clearer, not by changing its numbers, which he considers the right outcome for honest research. He sends reviewers a letter listing every change made.

Worked example

Yusuf answers critiques on power, comparator choice and mechanism with a power simulation, a labelled exploratory sensitivity analysis and a mechanism section, keeps the confirmatory result unchanged and narrows his conclusion.

Synthetic case. The critique and revisions are synthetic.

Errors and limits

Presenting exploratory reanalysis as confirmatory evidence hides researcher choices.

Exercises

• EX033 — Three critiques. Name the three reviewer points.
  Show marking criteria

  Rubric: 3 = power, comparator choice, missing mechanism;; 2 = all;; 1 = some;; 0 = none.
• EX034 — Power finding. What did the power simulation show?
  Show marking criteria

  Rubric: 3 = about thirty percent detection, far below eighty;; 2 = partly correct;; 1 = vague;; 0 = none.
• EX035 — Excluded comparator. Which comparator was excluded in sensitivity analysis?
  Show marking criteria

  Rubric: 3 = the water utility;; 2 = partly correct;; 1 = other;; 0 = none.
• EX036 — Sensitivity label. How is the sensitivity analysis labelled?
  Show marking criteria

  Rubric: 3 = exploratory;; 2 = partly correct;; 1 = confirmatory;; 0 = none.
• EX037 — Mechanism contents. What did risk-ranked patching change?
  Show marking criteria

  Rubric: 3 = category-based scoring, high systems first, weekly reports;; 2 = most;; 1 = few;; 0 = none.
• EX038 — Narrowed conclusion. What is the narrowed conclusion?
  Show marking criteria

  Rubric: 3 = a modest, uncertain reduction in one council;; 2 = partly correct;; 1 = vague;; 0 = none.