Chapter 6 — Peer critique and revision
Lesson map
Lesson 06 published. Map, example, EX033.
flowchart TD
N0["Learning outcomes"]
N1["6.1 Internal seminar"]
N2["6.2 Responding to reviewers"]
N3["6.3 Revising the procedure"]
N4["6.4 Keeping a decision log"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX033"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["6.1 Internal seminar"]
N2["6.2 Responding to reviewers"]
N3["6.3 Revising the procedure"]
N4["6.4 Keeping a decision log"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX033"]
[Learning outcomes]
|
v
[6.1 Internal seminar]
|
v
[6.2 Responding to reviewers]
|
v
[6.3 Revising the procedure]
|
v
[6.4 Keeping a decision log]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX033]
Learning outcomes
You will (1) present results for critique; (2) respond to challenges; (3) revise claims and methods.
6.1 Internal seminar
Ines presents the study at a department seminar. Colleagues question whether volunteer teams were simply more motivated, whether one quarter is long enough to show lasting change, and whether the codebook favours issues that the procedure is designed to find. She records each challenge and her answer, and marks those she cannot fully answer as limitations rather than defending them.
An internal seminar lets colleagues test the argument before outside reviewers do. Ines presents the design, results and limitations, then records questions she could not answer well. Those questions show where the thesis needs clearer explanation or further analysis, and preparing for them makes the eventual defense far less daunting.
6.2 Responding to reviewers
A workshop paper based on the baseline replication receives three reviews. One reviewer asks for the rule-size distribution and a sensitivity analysis excluding the two largest firewalls. Ines runs it, finds the association remains strong, and adds it to the paper. Another asks whether shadowing should count as an error when engineers use it deliberately as documentation; she adds a discussion and keeps the category, noting that undocumented shadowing is the main concern.
Each response cites the page where the change was made.
6.3 Revising the procedure
Partner engineers suggest that hit counts over ninety days miss rules for quarterly or yearly processes, such as financial year-end transfers. Ines changes the procedure to flag such rules for owner confirmation rather than marking them unused, and checks whether this changes earlier results. It reclassifies three rules but leaves the conclusions unchanged.
Feedback may show that parts of the review procedure were unclear or too slow. Ines revises the procedure, records each change with its reason, and keeps the original version, so readers know which version produced the results. Changes made after results are seen are labelled clearly as post hoc and are not presented as tested findings.
6.4 Keeping a decision log
Every change to codebook, procedure or analysis is entered in a dated decision log with the reason and its effect on results. The log is part of the thesis appendix, so examiners can see how the study evolved and that changes were not made to improve outcomes.
A decision log records every significant research choice with its date, the options considered, the choice made and the reason. Over a multi-year project memory fades, and examiners often ask why a particular choice was made. The log answers such questions with evidence, protects against unconscious rewriting of history, and makes the research easier for others to reproduce or extend.
Worked example
The decision log grows to twenty-three entries by the end of the critique phase, each linked to the analysis version it affected.
Synthetic case. Reviews and seminar are synthetic.
Errors and limits
Treating every critique as an attack leads to defensive writing; recording and answering critiques improves both study and trust.
Exercises
- EX033 — Motivation challenge. How should Ines respond to the motivated-volunteers critique? Rubric: 3 = acknowledge as limitation, use comparison and variables;; 2 = acknowledge;; 1 = dismiss;; 0 = none.
- EX034 — Sensitivity analysis. What is a sensitivity analysis here? Rubric: 3 = rerun excluding largest firewalls;; 2 = rerun;; 1 = vague;; 0 = none.
- EX035 — Deliberate shadowing. When might shadowing be deliberate, and how is it handled? Rubric: 3 = documentation purposes, flagged if undocumented;; 2 = deliberate;; 1 = vague;; 0 = none.
- EX036 — Periodic rules. Why might a ninety-day hit window misclassify rules? Rubric: 3 = quarterly or yearly processes;; 2 = missing rare use;; 1 = vague;; 0 = none.
- EX037 — Decision log. What does the decision log show examiners? Rubric: 3 = evolution and reasons, not outcome-driven;; 2 = history;; 1 = vague;; 0 = none.
- EX038 — Unanswered critiques. How should critiques Ines cannot answer be treated? Rubric: 3 = stated as limitations;; 2 = noted;; 1 = ignored;; 0 = none.
Synthesis
Structured critique leads to clearer claims, a sensitivity analysis and a better procedure.
Continue studying — chapter 06 of 08
← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →