Chapter 6 — Peer critique and revision | Doctoral research Security operations and blue team

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 06 published. Map, example, EX033.

flowchart TD
  N0["Learning outcomes"]
  N1["6.1 Sources of critique"]
  N2["6.2 The main critiques"]
  N3["6.3 Classifying severity"]
  N4["6.4 Responses"]
  N5["6.5 The response letter"]
  N6["6.6 Avoiding defensive revision"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX033"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["6.1 Sources of critique"]
  N2["6.2 The main critiques"]
  N3["6.3 Classifying severity"]
  N4["6.4 Responses"]
  N5["6.5 The response letter"]
  N6["6.6 Avoiding defensive revision"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX033"]

[Learning outcomes]
    |
    v
[6.1 Sources of critique]
    |
    v
[6.2 The main critiques]
    |
    v
[6.3 Classifying severity]
    |
    v
[6.4 Responses]
    |
    v
[6.5 The response letter]
    |
    v
[6.6 Avoiding defensive revision]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX033]

Learning outcomes

You will (1) seek and receive structured critique; (2) classify critiques by severity; (3) revise claims, analyses and text in response; (4) document responses transparently.

6.1 Sources of critique

Nadia's committee, an external examiner in training, Northgate's senior analysts and a workshop paper's anonymous reviewers all read the draft. Each sees different weaknesses. Each source looks at the thesis differently: the committee at method, the trainee examiner at structure, the senior analysts at realism and the workshop reviewers at the claim. Nadia logs every comment in one table with its source, so none is lost between drafts. The table also records which draft each change entered and whether the commenter saw the revised text, so Prof. Grant can check that each source had a chance to respond to Nadia's changes.

6.2 The main critiques

Reviewer 1: "The corpus is synthetic; benign triggers in real SOCs are messier." Reviewer 2: "The benign-history field may be doing all the work; you cannot separate the three fields." Reviewer 3: "Two missed attacks are a safety issue, not a footnote." A senior analyst: "Your alert cards lack the raw command line that we always look at first." The analyst's comment shows how practice can see what researchers miss: the cards were designed from the Contextual property, but analysts work from the raw event.

6.3 Classifying severity

Nadia sorts critiques into: threatens validity (Reviewer 2), changes conclusions (Reviewer 3), limits scope (Reviewer 1), improves realism (analyst). Validity threats are answered first. Answering validity threats first makes sense because they could change what the data mean. Reviewer 2's point led to the exploratory follow-up with single-field cards, while Reviewer 3's point changed how prominently the missed attacks were reported. The analyst's comment led to the raw command line being restored on all cards. Reviewer 1's point became a stated limit on realism, with a plan for an operational pilot.

6.4 Responses

To Reviewer 2 she runs an exploratory follow-up with four analysts and single-field cards; history alone gives most of the gain and both missed attacks — evidence that history context is powerful and risky. To Reviewer 3 she moves the harm to the abstract and recommends showing history only alongside a "why it could still be malicious" prompt. To Reviewer 1 she adds a limits section and releases the generator so others can test realism. The analyst's point is accepted for the next study.

6.5 The response letter

Each critique is quoted, followed by the change made (with page numbers) or a reasoned disagreement. Nadia's letter answers Reviewer 2 by describing the follow-up and labelling its result exploratory, so the reviewer can see that the confound was addressed without overstating it. Where she disagrees, as with a request to drop the synthetic corpus altogether, she explains why with evidence from the realism review and the ethical limits on real SOC data. Every change carries a page number. Prof. Grant reads the letter before it is sent, checking that each claimed change can be found on the page it cites.

6.6 Avoiding defensive revision

Grant reminds her that rebuttal without change rarely persuades; where a reviewer is wrong, show evidence. For Reviewer 3, rebuttal would have meant arguing that two attacks in many decisions were a small number. Nadia instead moved the missed attacks into the abstract and added a design recommendation, which persuaded the reviewer more than any argument could. Grant's reminder applies to her own instincts as much as to the reviewers' tone. The response letter shows the change for each critique that was accepted.

Worked example

Response excerpt: "R2.1 — We agree the fields are confounded. A follow-up (n = 4 analysts, 40 alerts each) with single-field cards suggests the benign-history field accounts for most of the improvement and for both missed attacks (reported in the critique chapter, labelled exploratory)."

Synthetic case. Critiques and follow-up are invented.

Errors and limits

A four-analyst follow-up is exploratory; it guides future work but cannot settle attribution.

Exercises

EX033 — Severity classes. Name the four severity classes used.
Show marking criteria
Rubric: 3 = validity, conclusions, scope, realism; 2 = three; 1 = one; 0 = none.
EX034 — Confounded fields. What was Reviewer 2's critique, and how was it answered?
Show marking criteria
Rubric: 3 = fields confounded; single-field follow-up, history drives gain and misses; 2 = partial; 1 = ignored; 0 = none.
EX035 — Safety framing. How was the missed-attack critique handled?
Show marking criteria
Rubric: 3 = moved to abstract; recommend counter-prompt with history; 2 = partial; 1 = footnote; 0 = none.
EX036 — Realism response. What answered the synthetic-corpus critique?
Show marking criteria
Rubric: 3 = limits section; release generator for testing; 2 = one; 1 = vague; 0 = none.
EX037 — Letter format. Describe a good response letter entry.
Show marking criteria
Rubric: 3 = quote, change with location or reasoned disagreement; 2 = partial; 1 = rebuttal only; 0 = none.
EX038 — Follow-up status. Why is the follow-up labelled exploratory?
Show marking criteria
Rubric: 3 = small, unplanned for confirmation; cannot settle attribution; 2 = partial; 1 = confirmatory; 0 = none.

Synthesis

Gather critique widely, rank it by threat to validity, revise with evidence and document every response.

Continue studying — chapter 06 of 08

← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →