Chapter 6 — Peer critique and revision | Doctoral research Threat intelligence and OSINT

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 06 published. Map, example, EX036.

flowchart TD
  N0["Learning outcomes"]
  N1["6.1 Inviting the hardest reader"]
  N2["6.2 ICD 203 as a lens"]
  N3["6.3 Point-by-point responses"]
  N4["6.4 Substantive versus cosmetic"]
  N5["6.5 Alternative hypotheses"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX036"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["6.1 Inviting the hardest reader"]
  N2["6.2 ICD 203 as a lens"]
  N3["6.3 Point-by-point responses"]
  N4["6.4 Substantive versus cosmetic"]
  N5["6.5 Alternative hypotheses"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX036"]

[Learning outcomes]
    |
    v
[6.1 Inviting the hardest reader]
    |
    v
[6.2 ICD 203 as a lens]
    |
    v
[6.3 Point-by-point responses]
    |
    v
[6.4 Substantive versus cosmetic]
    |
    v
[6.5 Alternative hypotheses]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX036]

Learning outcomes

You will (1) solicit adversarial critique of your own study; (2) apply ICD 203 tradecraft standards as a review lens; (3) respond to reviewers point by point; (4) distinguish revisions that change conclusions from those that improve presentation.

6.1 Inviting the hardest reader

A thesis committee is friendly by design. Doctoral candidates should also seek a red-cell reader: someone asked to find the weakest assumption. In CTI method research the classic weak points are circular outcome sources, corpus selection bias toward dramatic incidents, and confidence scales mapped after seeing the data.

Tomas's red-cell reader attacked a point the committee had not raised: confirmed claims are more likely to be followed up in public, which inflates apparent calibration for likely-true statements. The answer, a sensitivity analysis under two assumptions about unresolved claims, changed the abstract.

6.2 ICD 203 as a lens

ICD 203 asks analytic products to describe source quality, express uncertainty, separate information from assumptions and judgments, incorporate alternatives, show relevance, argue clearly, explain changes of judgment, aim for accuracy and use effective visuals. A research paper about CTI should meet the same bar it measures. Tomas runs his draft against that list and finds he never stated how his own conclusion would change if outcome sources were only half as independent as assumed.

His fix applies the ICD 203 standard on uncertainty to his own conclusion: the thesis now states how the calibration estimate moves if unresolved claims are treated first as unconfirmed and then as confirmed, using the two-assumption sensitivity analysis.

6.3 Point-by-point responses

Each reviewer comment receives a numbered reply: accepted with change (cite page), accepted partially (explain), or declined (explain with evidence). Silence on a comment reads as evasion.

Tomas's reply to the red-cell reader's follow-up-bias comment would be "accepted with change": it names the sensitivity analysis, the two assumptions about unresolved claims and the page where both ranges appear. A reader can check the reply against the thesis without re-reading the whole chapter. When a community partner asked him to drop a negative result, Tomas's reply was "declined", citing the pre-registered analysis plan that required reporting it; the wording change that partner also requested was "accepted with change".

6.4 Substantive versus cosmetic

A sensitivity analysis that moves the headline Brier from 0.17 to 0.21 is substantive and must reach the abstract. Fixing figure fonts is cosmetic. Candidates who bury substantive changes in an appendix lose trust.

Tomas's sensitivity analysis is that kind of change. Treating unresolved claims under two assumptions produced two ranges, and both went into the abstract, because a reader who saw only the more favourable one would misjudge how much the conclusion depends on follow-up behaviour. The figure-font fix goes in the change log and nowhere else, while the Brier shift is listed in the response letter as the first substantive change.

6.5 Alternative hypotheses

Analysis of competing hypotheses, discussed by Heuer, asks which evidence is inconsistent with each explanation rather than which is consistent with the favourite. A critique round is a good moment to list alternatives: for example, that apparent overconfidence reflects selective follow-up reporting rather than analyst miscalibration.

Tomas lists that alternative explicitly. Evidence inconsistent with it would be similar follow-up rates for confirmed and unconfirmed claims; the sensitivity analysis is his way of showing how much the conclusion changes if the alternative is true.

Worked example

The red-cell reader argues that confirmed claims are more likely to be followed up in public, inflating apparent calibration for likely-true statements. Tomas adds a sensitivity analysis that treats unresolved claims under two assumptions and reports both ranges in the abstract.

Synthetic case. The red-cell critique and sensitivity ranges are invented.

Errors and limits

Critique is not personal; nor is every comment correct. Defend with evidence, not seniority.

Exercises

EX036 — Red-cell target. Name two assumptions a red-cell reader should attack first in Tomas's design.
Show marking criteria
Rubric: 3 = two of outcome independence / corpus selection / post-hoc scale mapping / follow-up bias; 2 = one; 1 = typos; 0 = none.
EX037 — Missing standard. Which ICD 203 element did Tomas's draft fail before revision?
Show marking criteria
Rubric: 3 = expressing uncertainty/sensitivity of own conclusion to assumptions; 2 = vague uncertainty; 1 = visuals; 0 = none.
EX038 — Reply types. List the three acceptable categories of response to a reviewer comment.
Show marking criteria
Rubric: 3 = accepted with change, partially accepted, declined with reasons; 2 = two; 1 = ignore; 0 = none.
EX039 — Substantive test. A reanalysis moves Brier from 0.17 to 0.21. Where must that appear?
Show marking criteria
Rubric: 3 = abstract/main results + explanation; 2 = results only; 1 = appendix only; 0 = none.
EX040 — Competing story. State one alternative explanation for apparent overconfidence in the top bin.
Show marking criteria
Rubric: 3 = plausible alternative (selective follow-up, outcome-source bias, small k); 2 = vague; 1 = minimal attempt; 0 = none.
EX041 — ACH move. In Heuer's analysis of competing hypotheses, what evidence matters most?
Show marking criteria
Rubric: 3 = evidence inconsistent with hypotheses (disconfirming); 2 = diagnostic evidence vaguely; 1 = consistent evidence; 0 = none.
EX042 — Decline well. Write a one-sentence decline to a reviewer who asks for real victim names.
Show marking criteria
Rubric: 3 = decline citing ethics/TLP/data protection; offer aggregate alternative; 2 = decline only; 1 = comply; 0 = none.

Synthesis

Invite hostile reading, hold yourself to ICD 203, answer every comment, promote substantive changes to the front, and test alternatives by disconfirmation.

Continue studying — chapter 06 of 08

← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →