Chapter 10 — Research reading and critique
Lesson map
Lesson 10 published. Map, example, EX046.
flowchart TD
N0["Learning outcomes"]
N1["10.1 Why read research"]
N2["10.2 Reading a paper"]
N3["10.3 Example: data growth"]
N4["10.4 Example: corpora"]
N5["10.5 Example: similarity hashing"]
N6["10.6 Critique"]
N7["10.7 From reading to practice"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX046"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["10.1 Why read research"]
N2["10.2 Reading a paper"]
N3["10.3 Example: data growth"]
N4["10.4 Example: corpora"]
N5["10.5 Example: similarity hashing"]
N6["10.6 Critique"]
N7["10.7 From reading to practice"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX046"]
[Learning outcomes]
|
v
[10.1 Why read research]
|
v
[10.2 Reading a paper]
|
v
[10.3 Example: data growth]
|
v
[10.4 Example: corpora]
|
v
[10.5 Example: similarity hashing]
|
v
[10.6 Critique]
|
v
[10.7 From reading to practice]
|
v
[Worked example] --> [Exercise EX046]
Learning outcomes
You will (1) read forensic research critically; (2) assess methods, data and claims; (3) translate research into practice decisions; (4) identify gaps worth studying.
10.1 Why read research
Practice changes when evidence changes. Marta's team holds a monthly reading group, choosing papers relevant to current casework. For Marta's unit, research has practical stakes: a new method for memory analysis or triage may change what the team collects in the next incident. Choosing papers linked to current casework keeps discussion focused on decisions the team actually has to make. Each session ends with a note of what, if anything, should change. Attendance is recorded, and each examiner presents a paper at least once a year.
10.2 Reading a paper
Questions: What is the claim? What data were used — synthetic, real, how many? Is the method described well enough to repeat? Are error rates or limits reported? Who funded it? These questions apply equally to classic and recent papers, whatever their reputation. A claim tested only on synthetic data may not hold on Corvane's systems, and a method described too briefly to repeat cannot be validated locally. Error rates matter most, because a method used in court must have known limits.
10.3 Example: data growth
Garfinkel (2010) argued that the "golden age" of digital forensics was ending as data volumes, encryption and device diversity grew, and called for new research directions including standardised data and modular tools. The team asks whether their triage-then-image approach responds to that challenge. The team's answer is partly yes: triage-then-image handles volume by collecting selectively and imaging only significant hosts, as CF-21 showed with nine images out of 40 hosts. Encryption and device diversity remain hard, as the phone case's unreachable data showed. The discussion and any proposed change are recorded alongside the paper in the reading log.
10.4 Example: corpora
Garfinkel et al. (2009) described building standardised forensic corpora so results could be compared and tools tested. The team uses this to justify building its own synthetic test images for validation. Building synthetic images lets the unit validate tools such as the endpoint agent and the phone parser without exposing real personal data. Known contents mean the team can check what a tool finds against what is there. The same images also support the twice-yearly proficiency tests, which need cases with known answers.
10.5 Example: similarity hashing
Kornblum (2006) introduced context-triggered piecewise hashing; Roussev (2010) proposed similarity digests. The team discusses whether similarity matching could find ransomware variants across hosts, and what false-match rates would mean in practice. The question for Corvane is concrete: the ransomware binaries on different hosts might be small variants that exact hashes treat as unrelated. A similarity method could group them, but a false match could link an innocent file to the incident. The team decides to test on synthetic variants before adopting anything in casework.
10.6 Critique
Strong papers state limits; weaker ones generalise from one data set. The group writes a one-paragraph critique per paper. For Kornblum (2006), for instance, the critique notes that the method was demonstrated on specific data and that its behaviour on Corvane's file types would need testing. Each critique ends with whether the paper should affect practice and what local test would show it. The paragraph names the claim, the data, the main limit and the relevance to Corvane, so later readers can find it quickly. Critiques are filed with the paper in the reading log.
10.7 From reading to practice
A paper's method enters practice only after local validation. For similarity hashing this means Aiyana's test on synthetic variants first, then validation on the team's own corpus with measured false-match rates. Only then could it become a triage aid, and even then exact hashes remain the basis for identification. The open item in the programme review reflects that this is not yet complete. The validation result, with its data and error rates, is recorded before any procedure changes. A method that fails validation is noted as tried and rejected, so the team does not repeat the work.
Choosing papers
The group picks papers that address a current decision, alternating classic and recent work, and records why each was chosen.
Worked example
After reading Kornblum (2006), Aiyana tests similarity hashing on synthetic ransomware-like binaries modified in small ways. Exact hashes match none of the variants; similarity scores group them clearly. She proposes similarity matching as a triage aid, validated on the team's corpus, with exact hashes still used for identification.
Synthetic case. The test is invented.
Errors and limits
Older papers may describe tools or systems that have changed; check whether findings still hold.
Exercises
- EX046 — Reading questions. List four questions to ask of a paper. Rubric: 3 = claim, data, repeatability, limits/error rates, funding (any four); 2 = three; 1 = one; 0 = none.
- EX047 — Garfinkel 2010. Summarise Garfinkel's (2010) argument. Rubric: 3 = data growth, encryption, diversity end golden age; new research needed; 2 = partial; 1 = vague; 0 = none.
- EX048 — Corpora use. How does Garfinkel et al. (2009) support the team's practice? Rubric: 3 = justifies synthetic standard test images; 2 = partial; 1 = vague; 0 = none.
- EX049 — Similarity test. What did Aiyana's similarity test show? Rubric: 3 = exact hashes missed variants; similarity grouped them; 2 = partial; 1 = vague; 0 = none.
- EX050 — Practice rule. What must happen before a research method enters practice? Rubric: 3 = local validation; 2 = partial; 1 = vague; 0 = none.
Synthesis
Critical reading links research to practice, but only local validation earns a method its place.
Continue studying — chapter 10 of 12
← Chapter 09 (draft) · Course chapter index · Chapter 11 (draft) →