Chapter 6 — Peer critique and revision
Lesson map
Lesson 06 published. Map, example, EX033.
flowchart TD
N0["Learning outcomes"]
N1["6.1 The critique"]
N2["6.2 Response"]
N3["6.3 Second critique"]
N4["6.4 Revision"]
N5["6.5 Tone"]
N6["Further review"]
N7["Revision log"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX033"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["6.1 The critique"]
N2["6.2 Response"]
N3["6.3 Second critique"]
N4["6.4 Revision"]
N5["6.5 Tone"]
N6["Further review"]
N7["Revision log"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX033"]
[Learning outcomes]
|
v
[6.1 The critique]
|
v
[6.2 Response]
|
v
[6.3 Second critique]
|
v
[6.4 Revision]
|
v
[6.5 Tone]
|
v
[Further review]
|
v
[Revision log]
|
v
[Worked example] --> [Exercise EX033]
Learning outcomes
You will (1) respond to peer review; (2) revise claims; (3) strengthen limitations.
6.1 The critique
A workshop reviewer argues that the ring site's IT team knew it was being studied and may have worked harder, a Hawthorne effect. The critique is fair: the ring site's desktop team made a visible change, met Yusuf regularly and knew their results would be reported. Extra effort could have reduced both days to patch and failures, independent of the ring design. Yusuf records the critique in his response table and treats it as a threat to validity to address with evidence rather than argument.
6.2 Response
Yusuf checks help-desk staffing and overtime records: they did not differ between sites. He adds this check to the thesis and softens the claim to "consistent with" rather than "caused by". Prof. Solberg advised accepting the point openly, because equal staffing reduces the concern but cannot rule out other kinds of extra attention. The thesis therefore keeps the Hawthorne effect among the limitations, alongside the evidence that weakens it. The staffing comparison and its sources appear in an appendix, so examiners can check that the two sites were in fact alike.
6.3 Second critique
Another reviewer asks whether failures were defined consistently. Yusuf re-codes a random 20% of main-study tickets with a new coder; kappa remains above the pre-registered threshold. The new coder had not seen the original labels and used the revised codebook with the reboot rule. The 20% sample was drawn at random by script, with the seed recorded, so nobody could choose easy tickets. Keeping kappa above the threshold shows that the failure rate does not depend on one coder's habits. The re-coding sample and its kappa are in the archive.
6.4 Revision
The paper adds a limitations section on two sites, one sector and synthetic-structure data release. Each limitation names its effect on the claim: two sites limit generalisation to other trusts, one sector limits it to organisations with similar device constraints, and synthetic-structure release means readers can rerun the analysis but not inspect real tickets. Stating them in the paper prevents readers from extending the result further than the evidence supports. Each limitation also points to the follow-up study that would address it, such as more trusts or randomised start dates.
6.5 Tone
Responses thank reviewers and address each point with evidence. Yusuf replies to each reviewer in a table: the comment, the evidence gathered, the change made and its location in the paper. For the Hawthorne critique the reply cites the staffing records and the softened wording; for the coding critique it cites the re-coding sample and kappa. Where he disagrees, he explains why with evidence instead of dismissing the point, and the table goes into the archive. Reviewers saw the table first, so changes were easy to verify.
Further review
A third reviewer, a hospital IT manager, asks whether rings delayed urgent patches. Yusuf checks the record: two emergency updates during the study bypassed rings under the trust's emergency process, both reaching all devices within three days. He adds this to the paper, explaining that rings are a default, not a barrier to urgent action.
The revised paper is clearer about when the findings apply and how organisations should handle exceptions.
Revision log
Every change made in response to critique is recorded in a revision log with the reviewer's point, the evidence and the change, so examiners can trace how the argument evolved.
Second coder detail
The new coder was trained only on the codebook and had not seen earlier labels, so agreement reflects the codebook rather than shared habits. Disagreements were resolved by discussion and recorded.
Worked example
Prof. Solberg advises accepting the Hawthorne point openly rather than dismissing it, because the staffing check is reassuring but not conclusive.
Synthetic case. Synthetic.
Errors and limits
Defensive replies to critique weaken a thesis; evidence-based revisions strengthen it.
Exercises
- EX033 — Hawthorne. What was the Hawthorne critique? Rubric: 3 = ring IT team worked harder knowing they were studied; 2 = partial; 1 = vague; 0 = none.
- EX034 — Staffing check. What evidence addressed it? Rubric: 3 = staffing and overtime records equal; 2 = partial; 1 = vague; 0 = none.
- EX035 — Claim change. How was the claim softened? Rubric: 3 = consistent with; 2 = partial; 1 = vague; 0 = none.
- EX036 — Recoding. How was failure coding re-checked? Rubric: 3 = new coder on 20% sample; 2 = partial; 1 = vague; 0 = none.
- EX037 — Limitations. Which limitations were added? Rubric: 3 = two sites, one sector, synthetic-structure data; 2 = two; 1 = one; 0 = none.
- EX038 — Advice. What did Prof. Solberg advise? Rubric: 3 = accept openly; 2 = partial; 1 = vague; 0 = none.
Synthesis
Revise claims to match evidence after critique.
Continue studying — chapter 06 of 08
← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →