◇ OpenCyber Academy
Doctoral research · Chapter 6 · English

Chapter 6 — Peer critique and revision

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 6 — Peer critique and revision

Lesson map

Lesson 06 published. Map, example, EX033.

flowchart TD
  N0["Learning outcomes"]
  N1["6.1 Seeking critique"]
  N2["6.2 The critiques"]
  N3["6.3 Fatal or fixable"]
  N4["6.4 Revising claims"]
  N5["6.5 Change log"]
  N6["6.6 Practitioner guidance"]
  N7["Responding to reviewers"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX033"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["6.1 Seeking critique"]
  N2["6.2 The critiques"]
  N3["6.3 Fatal or fixable"]
  N4["6.4 Revising claims"]
  N5["6.5 Change log"]
  N6["6.6 Practitioner guidance"]
  N7["Responding to reviewers"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX033"]

[Learning outcomes]
    |
    v
[6.1 Seeking critique]
    |
    v
[6.2 The critiques]
    |
    v
[6.3 Fatal or fixable]
    |
    v
[6.4 Revising claims]
    |
    v
[6.5 Change log]
    |
    v
[6.6 Practitioner guidance]
    |
    v
[Responding to reviewers]
    |
    v
[Worked example] --> [Exercise EX033]

Learning outcomes

You will (1) invite structured critique of a supply-chain study; (2) separate fatal from fixable criticisms; (3) revise claims to match evidence; (4) record changes transparently.

6.1 Seeking critique

Yusuf submits a draft paper to a workshop and shares it with two practitioners at the Larchwood Foundation. He asks each reviewer to say what would make them reject the main claim. Asking reviewers what would make them reject the claim directs their attention to its weakest point instead of to presentation. The practitioners bring a different test: whether the guidance would help a maintainer facing a failing bot update. Yusuf logs every comment with its source in the change log before deciding how to respond.

6.2 The critiques

Reviewer 1: projects that adopt bots may already be more security-aware, so the bot may not cause the difference. Reviewer 2: exposure ends at the manifest change, but users may run old releases for months. Practitioner: the guidance should say what to do when tests fail, not just "use a bot". Each critique targets a different part of the argument: the first the causal reading, the second the definition of exposure, the third the usefulness of the guidance. Yusuf responds to each separately instead of answering them together.

6.3 Fatal or fixable

None is fatal. Reviewer 1's point limits the causal claim; Yusuf adds a within-project comparison of exposure before and after projects adopted a bot, which shows a smaller but still clear reduction (12 days). Reviewer 2's point defines the scope: the thesis measures producer-side exposure. The practitioner's comment was also fixable: the guidance gains a step for failing bot pull requests. None of the three required discarding data; each changed what the thesis claims, measures or recommends. The 12-day figure is reported beside the 21-day figure so readers can see both.

6.4 Revising claims

The headline changes from "bots cut exposure by 21 days" to "projects using bots had 21 days shorter median exposure; within projects, adopting a bot was followed by a 12-day reduction". The new headline names both comparisons and avoids saying that bots cause the reduction. Yusuf applies the same check to the abstract and the practitioner report, so the shorter claim appears everywhere. A reviewer who reads only the headline now gets the same scope as one who reads the method. The abstract now carries both comparisons' figures, matching the revised headline.

6.5 Change log

Every revision is listed with the reviewer comment that prompted it and where the text changed. The change log lets reviewers verify each response without rereading the whole paper. It also enters the archive, so examiners can trace the 12-day within-project comparison back to Reviewer 1's comment. One reviewer later asked to use it as a teaching example, which shows how much time it saves. Comments that led to no change are listed too, with the reason, so reviewers can see that nothing was ignored.

6.6 Practitioner guidance

The guidance adds: invest in tests for dependency updates; treat a failing bot pull request as a security task with an owner, consistent with SSDF RV.2. Both points come from the interviews and the p04 case: bot pull requests often failed because tests broke, and nobody owned the fix. The guidance does not claim that tests will remove all lag, since the interview counts include other reasons such as pinned transitive dependencies and lack of time. The step asks each project to name an owner for failing updates.

Responding to reviewers

Yusuf answers each comment point by point, quoting it, stating what he changed and where, or explaining politely why he did not. He avoids defensive language; disagreements are argued with evidence, such as the within-project analysis, rather than with assertions.

Worked example

The workshop accepts the revised paper. One reviewer notes the change log made re-review fast and asks to use it as an example for students.

Synthetic case. Reviews and figures are synthetic.

Errors and limits

Within-project comparisons can be confounded by time trends, such as ecosystem-wide tooling improvements.

Exercises

Synthesis

Ask what would sink the claim, fix what can be fixed and narrow what cannot.

Continue studying — chapter 06 of 08

← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →