Chapter 6 — Peer critique and revision

Lesson map

Lesson 06 published. Map, example, EX033.

flowchart TD
  N0["Learning outcomes"]
  N1["6.1 The critique"]
  N2["6.2 Testing alternatives"]
  N3["6.3 Avoiding p-hacking"]
  N4["6.4 Ethical reporting"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX033"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["6.1 The critique"]
  N2["6.2 Testing alternatives"]
  N3["6.3 Avoiding p-hacking"]
  N4["6.4 Ethical reporting"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX033"]

[Learning outcomes]
    |
    v
[6.1 The critique]
    |
    v
[6.2 Testing alternatives]
    |
    v
[6.3 Avoiding p-hacking]
    |
    v
[6.4 Ethical reporting]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX033]

Learning outcomes

You will (1) respond to peer review of a behaviour study; (2) revise analyses without p-hacking; (3) strengthen ethical reporting.

6.1 The critique

Mireille submits a paper and receives three reviews. One reviewer argues that more reporting might simply reflect more false alarms, burdening the security team. Another asks whether the thank-you messages, not the button, caused the effect. A third questions whether works council oversight changed staff behaviour by making them aware of the study. Prof. Aalto advises her to treat each comment as a hypothesis to test where possible, and to concede clearly where her data cannot answer. She writes a response letter that lists every comment with her action.

6.2 Testing alternatives

For false alarms, Mireille reports the share of reports coded benign before and after switching: it rose slightly, and the triage step kept handling time stable. For the mechanism question, she cannot separate button from feedback because they were introduced together, and she says so, proposing a factorial study as future work. For awareness of the study, she notes that announcements went to all departments at once, so awareness should not differ by switching time. Each response states what was pre-registered and what is exploratory.

6.3 Avoiding p-hacking

Reviewers suggest additional subgroup analyses by age and role. Mireille runs those that are reasonable but labels them exploratory and reports all of them, not just the interesting ones. She does not change her primary outcome or analysis. Herley and van Oorschot (2017) warn that security research often lacks the discipline of falsifiable, pre-specified claims; keeping the distinction between confirmatory and exploratory results is what lets readers trust the main finding. She also publishes a short table comparing the pre-registered plan with what was finally done, line by line, so that any deviation is visible and explained to readers.

6.4 Ethical reporting

A reviewer asks for more detail on harm. Mireille adds a section on how simulations were designed to avoid distress: no lures about pay cuts, bereavement or health; no lures impersonating trade unions; and a clear route for staff to complain. She reports two complaints received and how they were resolved. Ethical reporting in simulation research shows readers and organisations how to repeat the work without causing harm, which matters as much as the effect size. She also shares the full complaint procedure in an appendix.

Worked example

Mireille tests reviewers' alternative explanations, concedes the mechanism question, reports exploratory subgroups transparently and expands ethical reporting.

Synthetic case. Reviews and figures are synthetic; Herley and van Oorschot (2017) is real.

Errors and limits

Exploratory findings may be noise. Some reviewer concerns cannot be resolved with existing data.

Exercises

• EX033 — False-alarm test. How did Mireille address the false-alarm concern?
  Show marking criteria

  Rubric: 3 = benign share rose slightly; triage kept handling time stable;; 2 = data;; 1 = vague;; 0 = none.
• EX034 — Mechanism limit. Why can't she separate the button from feedback?
  Show marking criteria

  Rubric: 3 = they were introduced together;; 2 = confounded;; 1 = vague;; 0 = none.
• EX035 — Future design. What study did she propose for the mechanism question?
  Show marking criteria

  Rubric: 3 = a factorial study;; 2 = factorial;; 1 = vague;; 0 = none.
• EX036 — Subgroup handling. How were subgroup analyses reported?
  Show marking criteria

  Rubric: 3 = all reported and labelled exploratory;; 2 = exploratory;; 1 = selected;; 0 = none.
• EX037 — Excluded lure themes. Which lure themes were excluded to avoid distress?
  Show marking criteria

  Rubric: 3 = pay cuts, bereavement, health and union impersonation;; 2 = two;; 1 = vague;; 0 = none.
• EX038 — Complaints. How many complaints were reported?
  Show marking criteria

  Rubric: 3 = two, with their resolution;; 2 = two;; 1 = vague;; 0 = none.

Synthesis

Good revision tests alternatives, concedes limits, separates confirmatory from exploratory work and reports ethics openly.

Continue studying — chapter 06 of 08

← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →