Chapter 1 — Advanced research integrity and methods
Lesson map
Lesson 01 published. Map, example, EX001.
flowchart TD
N0["Learning outcomes"]
N1["1.1 Why integrity matters more in forensics"]
N2["1.2 Validation, verification, error rates"]
N3["1.3 Evidence for research"]
N4["1.4 Pre-registration"]
N5["1.5 Conflicts and tools"]
N6["1.6 Ethics review"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX001"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["1.1 Why integrity matters more in forensics"]
N2["1.2 Validation, verification, error rates"]
N3["1.3 Evidence for research"]
N4["1.4 Pre-registration"]
N5["1.5 Conflicts and tools"]
N6["1.6 Ethics review"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX001"]
[Learning outcomes]
|
v
[1.1 Why integrity matters more in forensics]
|
v
[1.2 Validation, verification, error rates]
|
v
[1.3 Evidence for research]
|
v
[1.4 Pre-registration]
|
v
[1.5 Conflicts and tools]
|
v
[1.6 Ethics review]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX001]
Learning outcomes
You will (1) apply research-integrity principles to digital forensic research; (2) distinguish validation, verification and error-rate estimation; (3) design studies that use synthetic or lawfully obtained evidence; (4) pre-register hypotheses and analysis plans.
1.1 Why integrity matters more in forensics
Forensic findings can decide liberty, employment and reputation. Research that overstates a tool's reliability can travel into courtrooms. Garfinkel's 2010 review of the field warned that digital forensics lacked the scientific foundations — standard corpora, error rates, reproducible methods — that other forensic sciences were being pushed to provide. A doctoral project should add to those foundations, not erode them. For Ines this has a concrete meaning. If she reports that header/footer carving recovers most JPEG-like files without stating fragmentation, an examiner could repeat that sentence in a report about a heavily fragmented drive, where the method may miss far more. Her thesis therefore ties every recall figure to its conditions and its interval.
1.2 Validation, verification, error rates
ISO/IEC 27041 frames validation as providing evidence that a method is fit for its stated purpose, including requirements, test design and acceptance criteria. Verification checks that a particular implementation or run behaves as the validated method says. An error rate states how often, under defined conditions, the method gives a wrong result. Ines's thesis must keep these three apart. In her study, validation is the pre-registered design showing that hash-scored recall measures what she claims. Verification is the check that each run used the archived carver, seeds and scorer. The error rate is one minus recall for a stated fragmentation level, reported with its 95% interval.
1.3 Evidence for research
Real case material carries privacy, legal and consent problems and is rarely shareable. Garfinkel, Farrell, Roussev and Dinolt (2009) argued for standardised forensic corpora so studies can be repeated. Ines builds synthetic disk images with known ground truth: every file, every fragment and every deletion is recorded at creation, so recall can be measured exactly. Synthetic images do not remove every problem. They answer how the method behaves on the generator's output, and Ines must still argue that the generator resembles real media, which is why a separate realism sub-study appears in her plan. Without that argument, a reader could dismiss the recall figures as properties of an artificial layout and nothing more.
1.4 Pre-registration
Before running the main study Ines writes hypotheses (H1: recall falls as fragmentation rises), primary measures (recall and precision per image), analysis (interval estimates, not just point values) and stopping rules. Changes later are allowed but logged with reasons. She deposits the document with Prof. Achebe, with a timestamped commit, before generating any main-study image. That order matters: if she saw the results first, she could pick the fragmentation levels or measures that make the carver look better or worse. The later deviation log is read against this document, so examiners can see which choices were fixed in advance.
1.5 Conflicts and tools
Tool vendors may fund research. Funding, tool versions and any relationship are declared. NIST's Computer Forensics Tool Testing programme shows the value of publishing test specifications and results openly. Ines receives no vendor funding, but her declaration still lists every tool by name and version, including the Python release used to build images and score output. A reader trying to reproduce a recall figure needs that list as much as the code, and Holloway Lab's policy asks for the declaration in the thesis front matter.
1.6 Ethics review
Even synthetic work needs ethics review if it involves people — for example, examiner studies. Participants consent; their performance data is pseudonymised. Ines's main study involves no participants, so her application to the university committee is short: it states that all images are generated, that no real media or personal data are used, and that no malware is handled. The planned future study of how examiners interpret partial recoveries would need full review, with consent forms and a plan for storing scores under pseudonyms. Writing the short application now gives the committee a record to compare against later.
Worked example
Ines's first draft claims her carver is "accurate". Her supervisor asks: accurate at what, under what conditions, against what ground truth, with what uncertainty? She rewrites the aim: "Estimate header/footer carving recall for JPEG-like files on synthetic images at 0%, 10% and 30% fragmentation, with 95% intervals."
Synthetic case. Holloway Lab, Ines and her images are invented.
Errors and limits
Synthetic images simplify real media. Results transfer only as far as the generator mimics reality; say so.
Exercises
- EX001 — Three terms. Distinguish validation, verification and error rate in forensic method research. Rubric: 3 = all three correct with example; 2 = two; 1 = one; 0 = none.
- EX002 — Why corpora. Why did Garfinkel et al. (2009) argue for standardised corpora? Rubric: 3 = repeatable, comparable studies without case-data barriers; 2 = partial; 1 = vague; 0 = none.
- EX003 — Ground truth. What advantage does a synthetic image give for measuring recall? Rubric: 3 = known ground truth of every file/fragment; exact recall; 2 = partial; 1 = speed; 0 = none.
- EX004 — Rewrite the aim. Rewrite "my tool is accurate" as a testable aim. Rubric: 3 = measure, conditions, ground truth, uncertainty; 2 = three; 1 = vague; 0 = none.
- EX005 — Pre-registration items. List four elements of a pre-registration. Rubric: 3 = hypotheses, measures, analysis, stopping/change rules; 2 = three; 1 = one; 0 = none.
- EX006 — Funding disclosure. What must be declared when a vendor funds tool research? Rubric: 3 = funding, relationships, tool versions; 2 = two; 1 = one; 0 = none.
- EX007 — Transfer limit. State the main limit on transferring synthetic-image results to real cases. Rubric: 3 = generator realism; state boundaries; 2 = partial; 1 = vague; 0 = none.
Synthesis
Forensic research must separate validation, verification and error rates, use shareable ground-truth evidence, pre-register and declare interests.