◇ OpenCyber Academy
Doctoral research · Chapter 6 · English

Chapter 6 — Peer critique and revision

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 6 — Peer critique and revision

Lesson map

Lesson 06 published. Map, example, EX033.

flowchart TD
  N0["Learning outcomes"]
  N1["6.1 Sources of critique"]
  N2["6.2 Organising"]
  N3["6.3 Common critiques"]
  N4["6.4 Legal relevance"]
  N5["6.5 Revising claims"]
  N6["6.6 Disagreement"]
  N7["6.7 Tracking"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX033"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["6.1 Sources of critique"]
  N2["6.2 Organising"]
  N3["6.3 Common critiques"]
  N4["6.4 Legal relevance"]
  N5["6.5 Revising claims"]
  N6["6.6 Disagreement"]
  N7["6.7 Tracking"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX033"]

[Learning outcomes]
    |
    v
[6.1 Sources of critique]
    |
    v
[6.2 Organising]
    |
    v
[6.3 Common critiques]
    |
    v
[6.4 Legal relevance]
    |
    v
[6.5 Revising claims]
    |
    v
[6.6 Disagreement]
    |
    v
[6.7 Tracking]
    |
    v
[Worked example] --> [Exercise EX033]

Learning outcomes

You will (1) solicit and organise critique of a forensic study; (2) respond point by point; (3) revise claims and analyses transparently; (4) handle critique about legal relevance.

6.1 Sources of critique

Supervisor, committee, lab peers, conference reviewers and practitioners. Practitioners ask whether results help casework; academics ask about method. Ines treats both kinds of question as useful. A practitioner asking whether 0.58 recall should change how carving results are reported points to the practitioner note she plans, while an academic asking about interval choice tests the method chapter. She keeps both in one table so neither group's comments are lost. Conference reviewers often ask for comparison with other carvers, which she answers by pointing to the declared scope and to the future work on reassembly.

6.2 Organising

Ines places every comment in a table: comment, category (method, data, interpretation, presentation), response, change made, location. The table makes progress visible to Prof. Achebe: each row shows whether a comment was accepted, partly accepted or answered with evidence, and where the thesis changed. Before each committee meeting she filters for rows still open, so no comment is forgotten. The final version of the table goes into the archive with the thesis, so examiners can trace each change back to the comment that prompted it.

6.3 Common critiques

"Synthetic images are unrealistic." Response: the realism sub-study and limits. "Only one file type." Response: accepted; listed as future work and a JPEG-like type justified by its prevalence in published casework. "Hash matching is too strict." Response: byte-overlap results added. Each response points to a place in the thesis where the change can be checked, such as the realism chapter or the byte-overlap table. A response that cannot point to such a place is treated as unfinished. Critiques about realism from three reviewers share one row.

6.4 Legal relevance

A practitioner asks whether the error rates would survive court scrutiny of scientific evidence. Ines answers that her rates apply only to the tested method and conditions; laboratories must validate their own tools and procedures under ISO/IEC 27041-style processes. She adds a short paragraph on legal relevance to her discussion chapter. It explains that a published error rate for a method under declared conditions is one input to a laboratory's validation, not a substitute for it, and it names the conditions that a laboratory would need to match before relying on her figures.

6.5 Revising claims

The abstract originally said "carvers fail on fragmented files". Revised: "Header/footer carving recall fell from 0.96 to 0.58 as fragmentation rose from 0% to 30% on synthetic images." The revision replaced a sweeping claim with one tied to a method, a range of conditions and a dataset. Ines checks every claim in the abstract against the results table before submission, and any sentence without a matching row is cut. The same check is repeated for the conclusion and for the practitioner note, because claims quoted out of context usually come from those places.

6.6 Disagreement

When Ines disagrees, she explains with evidence and offers a compromise, such as an added sensitivity analysis. One reviewer wanted her to drop the 10% condition because its interval overlaps both neighbours. Ines kept it, explaining that the overlap is itself a finding about sample size, and offered to report how many files per condition would separate the intervals. The reviewer accepted the compromise, and the comment table records the exchange. She never changes a result to settle a disagreement; only wording, analysis or scope can change, and each change is logged.

6.7 Tracking

All revisions are committed with messages referencing the comment table. Each commit message names the row number of the comment it answers, for example a change to the abstract answering a committee remark. A reader of the repository can therefore follow any change back to the person and reason that prompted it. Ines also tags the version sent to each round of reviewers, so comments are always read against the text they were made on. When a comment leads to no change, the row records why, so the absence of a commit is also explained.

Practitioner workshop

Ines presents draft results at a workshop of laboratory examiners. They ask for results per file size and for guidance on when to suspect fragmentation in casework. She adds a size breakdown and a short practitioner note, keeping both within the evidence her study supports, and thanks participants in the acknowledgements without naming their laboratories.

Worked example

A reviewer suspects the scorer favours the baseline. Ines publishes the scorer separately and invites the reviewer to run it on their own images. The reviewer reproduces her numbers and withdraws the concern.

Synthetic case. The review is invented.

Errors and limits

Some critiques cannot be resolved within a thesis; list them openly.

Exercises

Synthesis

Organise critique, respond with evidence, narrow claims to what was measured and make tools available for independent checking.

Continue studying — chapter 06 of 08

← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →

Worked answers

EX033. Comment, category, response, change made and location in the thesis.

EX034. She pointed to the realism sub-study and restated that conclusions are limited to the properties it measured.

EX035. Her rates describe only the tested method on synthetic images; each laboratory must validate its own tools and procedures.

EX036. "Carvers fail" overgeneralised; the revision states the measured recall drop and its conditions.

EX037. She released the scorer; the reviewer ran it on independent images, reproduced the results and withdrew the concern.

EX038. Record them openly as limitations or future work in the thesis.