Chapter 6 — Peer critique and revision
Lesson map
Lesson 06 published. Map, example, EX033.
flowchart TD
N0["Learning outcomes"]
N1["6.1 Seeking critique"]
N2["6.2 Objections received"]
N3["6.3 Responding"]
N4["6.4 Claim revision"]
N5["6.5 Recording changes"]
N6["Group critique"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX033"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["6.1 Seeking critique"]
N2["6.2 Objections received"]
N3["6.3 Responding"]
N4["6.4 Claim revision"]
N5["6.5 Recording changes"]
N6["Group critique"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX033"]
[Learning outcomes]
|
v
[6.1 Seeking critique]
|
v
[6.2 Objections received]
|
v
[6.3 Responding]
|
v
[6.4 Claim revision]
|
v
[6.5 Recording changes]
|
v
[Group critique]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX033]
Learning outcomes
You will (1) invite structured critique of the study; (2) separate valid objections from preferences; (3) revise claims and analyses; (4) record changes transparently.
6.1 Seeking critique
Amara presents to the research group and sends the draft paper to two external reviewers: a platform engineer and a measurement researcher. She asks each to identify the weakest claim and the weakest analysis.
The two reviewers cover different weaknesses: a platform engineer knows how teams react to blocked deployments, while a measurement researcher knows how outcome definitions can hide effects. Asking each for the weakest claim and the weakest analysis invites specific objections rather than general praise. Both receive the exposure definition, including the rule that a blocked deployment counts as zero exposure, which is where both later aim their objections.
6.2 Objections received
The platform engineer argues that blocking deployments may push teams to request broad exceptions, moving risk rather than removing it. The measurement researcher notes that blocked deployments counted as zero exposure could hide workarounds such as running the same workload in an unenforced namespace.
Both objections target the zero-exposure assumption for blocked deployments. If teams answered blocks with broad exceptions or by moving workloads to unenforced namespaces, the true exposure under admission would be higher than measured, and the 14.5-hour difference would overstate the effect. Each objection names a pattern that can be searched for in configuration history.
6.3 Responding
Amara checks both. Exception scope is analysed: 9 approved exceptions, all narrow to one workload. For workarounds, she searches the configuration history for workloads that were blocked and appeared within a day in another namespace; she finds two, both in a development namespace outside the study, and adds a sensitivity analysis counting them as exposed. The median difference stays 14.5 hours in the synthetic data.
The two workarounds were in a development namespace outside the study, so counting them as exposed tests the worst case. That the median difference stays the same shows that the result does not depend on the zero-exposure assumption for these cases. The sensitivity analysis is reported in the thesis even though it changes nothing, because the reader needs to see that the objection was tested.
6.4 Claim revision
The original claim "admission control eliminates exposure" becomes "admission enforcement reduced median exposure by about 14 hours, with a small number of exceptions and possible workarounds measured separately".
The revised claim keeps the measured size of the effect and adds what qualifies it. "Eliminates" could be refuted by a single workaround; "reduced median exposure by about 14 hours" with exceptions and workarounds measured separately is what the data support. The second reviewer's objection is answered in the claim itself rather than in a footnote.
6.5 Recording changes
A revision log lists each critique, the response and any change in text or analysis, kept in the archive.
For Amara's thesis the log has entries for both external reviewers' objections, the exception-scope analysis, the workaround sensitivity analysis, and the reviewer who preferred a different statistical test, where both tests agreed and the difference was recorded as preference. Keeping the log in the archive lets examiners see that each critique was answered with evidence. The workaround search, for example, appears with the two development-namespace workloads it found and the sensitivity analysis it added.
Group critique
The research group meeting adds a practical point: the exception process itself needs measuring, since slow approvals push teams toward workarounds.
Worked example
One reviewer prefers a different statistical test. Amara explains why the bootstrap fits a skewed, small sample and adds the alternative test as a supplementary result; both agree. The disagreement is recorded as a matter of preference, not a flaw.
Synthetic case. Reviewers and results are synthetic.
Errors and limits
Absorbing every suggestion can blur the paper; respond to all, change only what improves validity.
Exercises
- EX033 — Reviewers asked. What did Amara ask reviewers for? Rubric: 3 = weakest claim and analysis; 2 = partial; 1 = vague; 0 = none.
- EX034 — Workaround objection. What was the measurement objection? Rubric: 3 = zero exposure may hide workarounds; 2 = partial; 1 = vague; 0 = none.
- EX035 — Workarounds found. How many workarounds were found? Rubric: 3 = two in development namespace; 2 = partial; 1 = vague; 0 = none.
- EX036 — Revised claim. How was the claim revised? Rubric: 3 = reduced median ~14h with exceptions noted; 2 = partial; 1 = vague; 0 = none.
- EX037 — Test preference. How was the test preference handled? Rubric: 3 = alternative as supplementary; 2 = partial; 1 = vague; 0 = none.
- EX038 — Revision log. What does the revision log contain? Rubric: 3 = critique, response, change; 2 = two; 1 = one; 0 = none.
Synthesis
Critique strengthens claims when answered with evidence.
Continue studying — chapter 06 of 08
← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →