Chapter 7 — Peer critique and revision of claims

Lesson map

Lesson 07 published. Map, example, EX039.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["7.1 Why critique is part of the method"]
  N2["7.2 Organizing critique before submission"]
  N3["7.3 Classifying critiques"]
  N4["7.4 Reproducing contested analyses"]
  N5["7.6 Negative and partial results"]
  N6["7.7 Writing the response to reviewers"]
  N7["7.8 Post-publication critique and correction"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX039"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["7.1 Why critique is part of the method"]
  N2["7.2 Organizing critique before submission"]
  N3["7.3 Classifying critiques"]
  N4["7.4 Reproducing contested analyses"]
  N5["7.6 Negative and partial results"]
  N6["7.7 Writing the response to reviewers"]
  N7["7.8 Post-publication critique and correction"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX039"]

[Learning outcomes and connection]
    |
    v
[7.1 Why critique is part of the method]
    |
    v
[7.2 Organizing critique before submission]
    |
    v
[7.3 Classifying critiques]
    |
    v
[7.4 Reproducing contested analyses]
    |
    v
[7.6 Negative and partial results]
    |
    v
[7.7 Writing the response to reviewers]
    |
    v
[7.8 Post-publication critique and correction]
    |
    v
[Worked example] --> [Exercise EX039]

Learning outcomes and connection

By the end of this chapter the learner will be able to:

1. Organize adversarial review of one's own work before submission, including red-team reading and adversarial collaboration.
2. Classify a critique as conceptual, methodological, analytical, reproducibility-related or presentational, and choose the right response.
3. Reproduce a contested analysis under a critic's alternative definition and report both versions.
4. Revise a contribution statement so that it claims exactly what survives critique.
5. Report negative and partial results so that they are useful to others.

Chapters 5 and 6 produced a two-context result with a plausible boundary condition. Before dissemination, those claims must survive people trying to break them. Chapter 8 then assembles what survives into the thesis and its archive.

7.1 Why critique is part of the method

Peer review is often experienced as an obstacle after the work is done. In research practice it is better understood as a method step: independent critics test whether the inferential chain from observation to claim holds. Munafò et al. (2017) list evaluation measures — including pre-publication and post-publication review and the use of independent methodological support — among practices intended to make research more reliable. The Registered Reports format moves the most consequential review — of question and method — before data collection, so that critique can change the design rather than only the narrative.

Incident-response research needs critique for a specific reason: authors are often close to practice, partner organizations, or the artifact under test. Chapter 1 (§1.3.4) described structural safeguards against conflicted evaluation; critique by people who do not share the author's commitments is another.

7.2 Organizing critique before submission

• Red-team reading. Ask two or three colleagues to read the thesis chapter or paper as a hostile reviewer, with a brief: "Find the weakest link between observations and the main claim." Give them the protocol, deviation log and analysis code, not only the manuscript.
• Methods review. Ask a methodologist outside the field to review measurement and analysis; ask a practitioner outside the partner organization to review scenario realism and practical relevance.
• Reproduction by a third party. Ask someone to rerun the analysis from the archive (Chapter 8) without help. Every question they ask reveals missing documentation.
• Adversarial collaboration. When a critic holds a competing theory (Chapter 2 §2.3), invite them to agree in advance on a test whose outcomes would favor one account or the other. The agreement is recorded before data are examined.
• Pre-mortem. Imagine the paper has been retracted two years from now. Write down the most likely reasons. Each reason is a check to perform now.

7.3 Classifying critiques

Lesson table; scroll horizontally to read all columns →

In plain terms: Different critiques need different responses: a measurement critique calls for reanalysis, a scope critique for narrower wording, a misreading for clearer text.

Three responses are always available: concede (the critic is right; revise), defend (the critic is mistaken; explain why with evidence) and test (the question is empirical; run the analysis the critic proposes and report the result). The weakest response is to rephrase without changing anything.

7.4 Reproducing contested analyses

When a critic proposes an alternative definition or analysis, the default is to run it and report it, unless it is clearly invalid. Report the original preregistered analysis as primary, the critic's analysis as a sensitivity or robustness analysis, and explain any divergence. Never replace the primary analysis silently; never suppress the alternative because it is less favorable. If the alternative analysis reveals that the original claim depended on one contestable choice, the claim should be narrowed.

For the SHA study, a critic might recode errors under a broader definition of projection error; the raw rater sheets kept under §5.4 make that rerun possible.

7.5 Worked example: external reviewers challenge the main study's outcome definition

Synthetic. An external reviewer of the Handoff Study writes:

"Your primary outcome counts 'treating missing telemetry as negative evidence' as a comprehension error. But the SHA's T-gaps field tells participants which telemetry is missing. Of course they then make fewer such errors. This is not evidence that SHA improves comprehension; it is evidence that participants read the form. What happens to your effect if you exclude gap-misread errors?"

Classification. Conceptual and analytical: it challenges whether the outcome measures the construct claimed ("comprehension") and proposes a testable alternative.

Response plan. Partly defend: transmitting gap information is the proposed mechanism (Chapter 3 §3.2, link 4), and receiving responders in real handoffs do misread gaps, so reducing those errors is a legitimate decision-quality improvement. Partly concede: the primary outcome bundles gap-misreads with other comprehension and projection errors, so the claim "SHA improves comprehension" may overstate a narrower effect. Test: recode the synthetic decisions into gap-misread errors and all other comprehension/projection errors, and report both.

"""Chapter 7 offline lab: recode the Chapter 5 synthetic outcome under a reviewer's alternative definition."""
from itertools import product
from statistics import mean, stdev

# participant: ((standard_gap_misread, standard_other_cp), (sha_gap_misread, sha_other_cp))
SPLIT = {
    "P01": ((2, 3), (0, 3)), "P02": ((1, 3), (1, 3)), "P03": ((2, 4), (0, 3)),
    "P04": ((1, 2), (0, 2)), "P05": ((3, 4), (1, 3)), "P06": ((1, 3), (0, 3)),
    "P07": ((1, 4), (1, 4)), "P08": ((2, 4), (1, 3)), "P09": ((0, 2), (0, 2)),
    "P10": ((2, 3), (0, 2)), "P11": ((1, 3), (0, 3)), "P12": ((2, 4), (1, 4)),
}
T_CRIT_DF11 = 2.201

def summarize(label, d):
    m, s = mean(d), stdev(d)
    h = T_CRIT_DF11 * s / len(d) ** 0.5
    obs = abs(sum(d))
    p = sum(1 for sg in product((1, -1), repeat=len(d))
            if abs(sum(x * y for x, y in zip(d, sg))) >= obs) / 2 ** len(d)
    print(f"{label:28s} mean_diff={m:.3f} 95%CI=({m - h:.3f}, {m + h:.3f}) exact_p={p:.4f}")

if __name__ == "__main__":
    total = [sum(s) - sum(h) for s, h in SPLIT.values()]
    gap = [s[0] - h[0] for s, h in SPLIT.values()]
    other = [s[1] - h[1] for s, h in SPLIT.values()]
    summarize("original outcome (all c+p)", total)
    summarize("gap-misread errors only", gap)
    summarize("other c+p errors only", other)
    print(f"share of reduction from gap-misreads: {sum(gap) / sum(total):.2f}")

Expected output (verified with Python 3):

original outcome (all c+p)   mean_diff=1.417 95%CI=(0.677, 2.157) exact_p=0.0039
gap-misread errors only      mean_diff=1.083 95%CI=(0.580, 1.587) exact_p=0.0039
other c+p errors only        mean_diff=0.333 95%CI=(0.020, 0.646) exact_p=0.1250
share of reduction from gap-misreads: 0.76

Interpretation. About three-quarters (0.76) of the reduction comes from gap-misread errors. The reduction in other comprehension and projection errors is small (0.33 over three scenarios, about 0.11 per scenario). Its t-interval barely excludes zero, while the exact sign-flip test gives p = 0.125 — with only four non-zero differences out of twelve, the evidence for any broader effect is weak and method-dependent. The reviewer's challenge therefore changes the claim substantially.

Revised contribution statement.

Lesson table; scroll horizontally to read all columns →

In plain terms: The claim got smaller and more precise after critique; that is a gain, because practitioners now know where it applies.

The revised claim is narrower and more useful: it tells practitioners exactly what the artifact does, and it tells researchers which part of the original mechanism (ruled-out hypotheses, expectations) remains untested. The response letter thanks the reviewer, reports the new analysis, and revises abstract, discussion and conclusions accordingly.

7.6 Negative and partial results

A negative result — no effect, or an effect smaller than the SEOI — is informative when the study was adequately powered, measured validly and preregistered. Chapter 3's failure conditions (F3–F6) were written to make such results interpretable. Report them with the same structure as positive results: estimate, interval, relation to SEOI, mechanism checks and limitations. Distinguish clearly between "evidence of absence" (a precise null excluding the SEOI) and "absence of evidence" (an imprecise result). Archive negative results even if a venue declines them; Chapter 8's archive is designed to hold them.

7.7 Writing the response to reviewers

A good response document has, for each comment: the comment quoted; the classification; the response (concede, defend, test); what changed, with locations; and any new analysis, with code and output referenced. Keep the tone factual. Where you defend, give evidence, not assertion. Where you concede, do not hedge the revision into meaninglessness.

Suppose a reviewer argues that the SEOI of 0.5 errors per scenario was chosen to make the study feasible. The response quotes the cost–benefit anchoring from §3.13, reports the precision the sample actually achieved and, if the reviewer is right, narrows the claim. Each response names the page or file where the change can be checked.

7.8 Post-publication critique and correction

Critique continues after publication. If an error is found — a coding mistake, an archive that does not reproduce a table, a misdescribed sample — correct it publicly, through the venue's correction process and in the archive's change log. Corrections are an integrity strength. A thesis should state how readers can report problems and how corrections will be handled (Chapter 8).

For the SHA study, a correction might concern a scenario whose error count was mis-adjudicated. The change log would record the old and new values, the script that recomputes the primary estimate and whether the interval's relation to the SEOI changed.

7.9 Critique as structured adversarial review

Section 7.2 described organizing critique before submission. At doctoral level it helps to treat internal critique as a designed activity with roles, inputs and outputs, much like a red-team exercise in security.

Roles. Assign at least three critics with different vantage points: a methodologist (design, measurement, statistics), a practitioner (realism of scenarios, plausibility of the mechanism, operational cost), and an ethicist or data steward (consent, data handling, dual use). Ask each to produce their strongest objections independently before any discussion, to avoid convergence on the first objection voiced.

Inputs. Give critics the protocol, the pre-registration, the analysis code, the de-identified data or synthetic fixture, and the draft results — but ask the methodologist to read the protocol and code before the results, so that critique of the design is not shaped by the findings.

Outputs. A numbered objection list, each with a type (§7.3), severity (would it change the main claim, a secondary claim, or only wording?), and a suggested test. The candidate then responds in the format of §7.7, as if to a journal.

The practice exploits a finding already discussed in Chapter 5: Silberzahn et al. (2018) showed that independent analysts given the same data and question can reach materially different estimates through reasonable choices. A critic who reanalyzes the data independently, without seeing the candidate's code, is a one-person version of that design. If their estimate agrees, robustness is demonstrated; if not, the discrepancy localizes a fragile choice.

7.10 Common critiques of incident-response studies and how strong responses look

Experience with security user studies and with the pitfalls catalogued by Arp et al. (2022) suggests a recurring set of critiques. Anticipating them is cheaper than answering them after review.

Lesson table; scroll horizontally to read all columns →

In plain terms: Weak responses defend; strong responses show evidence or narrow the claim. The left column is predictable enough that a candidate can prepare the right column before submission.

The blinding check in the third row is inexpensive and seldom reported. After rating, ask each rater to guess the condition for each decision. If guessing accuracy is near chance, blinding held. If it is well above chance, report it and examine whether rater scores differ between correctly and incorrectly guessed items.

7.11 When critique concerns unfalsifiable claims

Herley and van Oorschot (2017) criticized parts of security research for claims that cannot be refuted by any observation. A thesis is vulnerable to this critique whenever its claims are hedged so heavily that no result could contradict them — for example, "structured handoffs can improve situational understanding in some contexts." The repair is to restore content: state the population, the outcome, the expected size, and the result that would count against the claim. Paradoxically, a more falsifiable claim is more defensible, because reviewers can see exactly what evidence supports it.

A useful self-test: for each claim in the abstract, write down an observation that would show it to be false. If none exists, the claim is not yet scientific; if the observation is one the study could have made and did not, the claim needs that study.

7.12 Registered reports and pre-publication review of design

Section 7.8 discussed correction after publication. The complementary approach moves critique earlier. Under the registered-report model described by the Center for Open Science, a study's introduction, methods and analysis plan are peer reviewed before data are collected, and a journal can commit in principle to publish the results if the protocol is followed. For a doctoral program, the same logic can be applied internally: the committee reviews and signs off the protocol, which then becomes the standard against which the final analysis is checked. Critique received at that stage changes the design rather than requiring the candidate to defend a flawed one.

The candidate should record how protocol-stage critique changed the design, because examiners value evidence that critique was used, and because it distinguishes design changes made before data from those made after.

7.13 Responding to practitioner critique

Incident-response research has two audiences: researchers and practitioners. Practitioner critique often arrives in a different register — "this would never work on a night shift with three people" — and is easy to dismiss as anecdote. It should instead be translated into a testable claim about a moderator or a boundary: with staffing below some level, senders lack time to complete the SHA, and completeness drops. That claim can be checked in field data (completeness of SHA fields by staffing level) or added to the transportability memo. Translating critique into hypotheses respects the critic and strengthens the thesis.

When practitioner and researcher critiques conflict — a methodologist wants more control, a practitioner wants more realism — the conflict is usually a trade-off between internal and external validity. The candidate's job is not to satisfy both but to state which the study prioritized, why, and what the follow-up study would prioritize instead.

7.14 Critique of AI-assisted extensions

If the thesis includes or proposes model-assisted handoffs (§3.15), anticipate critiques specific to that component:

• Version dependence. Results may not hold for later model versions. Record the exact version and settings; present results as conditional on them.
• Benchmark contamination. Public scenarios similar to the evaluation set may have appeared in a model's training data. Use privately written scenarios and say so.
• Evaluation by another model. If a model was used to score outputs, critics will ask how its scores relate to human judgment. Report agreement on a human-rated subset, or avoid model-based scoring for primary outcomes.
• Reliance effects. The reliance literature (Parasuraman and Manzey, 2010; Lee and See, 2004) predicts over-reliance; reviewers will ask whether the design included model errors. If not, the claim must be limited to cases where the model is correct, which is not a deployable claim.

Common errors and limits

• Treating reviewers as opponents to be persuaded rather than as tests to pass.
• Replacing the primary analysis with a critic's alternative, or ignoring the alternative.
• Revising wording without changing claims.
• Calling an imprecise null "no effect."
• Asking only friendly colleagues for review.
• Limit: critique cannot rescue a study whose raw data or design are inadequate; it can only reveal that.

Exercises — chapter 7

EX039 — Pre-mortem. Objective: anticipate failure. Inputs: the Handoff Study as of Chapter 6. Constraints: at least six plausible retraction or major-correction reasons. Steps: list reasons; for each, name the check that would detect it now. Expected evidence: table. Rubric: 3 = six or more specific reasons with concrete checks; 2 = some generic; 1 = vague; 0 = none.

EX040 — Critique classification and response. Objective: respond appropriately. Inputs: five synthetic reviewer comments: (a) "Raters could tell SHA answers by their bullet format"; (b) "You should use a mixed model"; (c) "The abstract says 'responders' but participants were volunteers from two teams"; (d) "I could not run your analysis: missing file decisions_v3.csv"; (e) "Situation awareness theory is outdated." Constraints: classify each and choose concede, defend or test. Steps: (1) classify each critique using §7.3; (2) decide accept, partially accept or rebut; (3) name the analysis or text change that follows; (4) draft the reply. Expected evidence: table with response drafts. Rubric: 3 = correct classifications and proportionate responses, (e) answered with evidence about predictions, not dismissal; 2 = one weak; 1 = defensive throughout; 0 = none.

EX041 — Contested analysis. Objective: reproduce a critic's alternative. Inputs: the script in §7.5. Constraints: offline. Steps: run; then compute the effect on gap-misread errors as a share of the standard condition's gap-misreads (relative reduction); write 150 words on what the alternative analysis does and does not show. Expected evidence: outputs, relative reduction and paragraph. Rubric: 3 = exact outputs, relative reduction correct (≈ 0.72), balanced interpretation; 2 = outputs correct, interpretation one-sided; 1 = one value wrong; 0 = none.

EX042 — Revised contribution statement. Objective: narrow a claim honestly. Inputs: Chapters 5–7 results. Constraints: ≤ 150 words; integrate the boundary from Chapter 6 and the narrowing from §7.5. Steps: (1) quote the Chapter 3 contribution statement; (2) mark each phrase that later evidence weakened; (3) rewrite with the narrowed scope; (4) add one sentence saying what was dropped and why. Expected evidence: statement. Rubric: 3 = narrowed to gap-misread mechanism, boundary noted, untested elements named; 2 = one omission; 1 = unchanged; 0 = none.

EX043 — Negative result report. Objective: report a null usefully. Inputs: suppose the KFL direct replication had found −0.05 (95% CI −0.40 to 0.30) per three scenarios. Constraints: use SEOI 1.5 per three scenarios. Steps: classify (evidence of absence or absence of evidence); write a 150-word results paragraph; state the implication for the claim. Expected evidence: paragraph. Rubric: 3 = precise null correctly identified, claim implications drawn; 2 = classification right, implications thin; 1 = "failed to replicate" only; 0 = none.

EX044 — Response letter excerpt. Objective: write a professional response. Inputs: the reviewer comment in §7.5. Constraints: 250–400 words; include classification, response, changes and new analysis. Steps: (1) quote the reviewer point verbatim; (2) answer in one sentence; (3) give the evidence and its location; (4) state the manuscript change, with page or section. Expected evidence: letter excerpt. Rubric: 3 = partly defend, partly concede, test reported, concrete changes listed; 2 = no concrete changes; 1 = purely defensive; 0 = none.

Chapter synthesis

Critique is the method step that tests the chain from observation to claim with people who do not share the author's commitments. Organized red-team reading, third-party reproduction and adversarial collaboration find weaknesses before publication; concede, defend or test are the honest responses. In the Handoff Study, a reviewer's challenge to the outcome definition showed that most of the effect came from transmitting telemetry gaps, and the contribution was narrowed accordingly. Chapter 8 assembles the narrowed claim, its evidence and its artifacts into a thesis and reproducibility archive.