Chapter 33 — Research methods II
Lesson map
Lesson 33 published. Map, example, EX097.
flowchart TD
N0["Learning outcomes"]
N1["33.1 From question to design"]
N2["33.2 Ground truth"]
N3["33.3 Criteria"]
N4["33.4 Confounders"]
N5["33.5 Analysis"]
N6["33.6 Ethics"]
N7["33.7 Limitations"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX097"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["33.1 From question to design"]
N2["33.2 Ground truth"]
N3["33.3 Criteria"]
N4["33.4 Confounders"]
N5["33.5 Analysis"]
N6["33.6 Ethics"]
N7["33.7 Limitations"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX097"]
[Learning outcomes]
|
v
[33.1 From question to design]
|
v
[33.2 Ground truth]
|
v
[33.3 Criteria]
|
v
[33.4 Confounders]
|
v
[33.5 Analysis]
|
v
[33.6 Ethics]
|
v
[33.7 Limitations]
|
v
[Worked example] --> [Exercise EX097]
Learning outcomes
You will (1) design a comparative study; (2) define evaluation criteria; (3) control confounding factors; (4) analyse and report results with limitations.
33.1 From question to design
Leah's comparative project asks: "On a synthetic Windows image with known planted artefacts, how completely do two freely available forensic tools report persistence and execution artefacts?"
The question fixes the scope: two tools, one synthetic image, two artefact types. Using planted artefacts means she knows the right answer in advance, which is the only way to measure completeness. A study on a real image could compare tools against each other but not against truth, since nobody would know what was really there.
33.2 Ground truth
She builds a synthetic image with a documented list of planted artefacts: files, run keys, prefetch entries, scheduled tasks. Ground truth is fixed before running either tool.
Fixing ground truth first prevents her from adjusting the list after seeing tool outputs, which would bias the results. Each planted item is documented with its location and expected appearance. The list is stored with a hash and a date, so a reviewer can confirm it was not changed during the study. Any operating-system defaults that appear on the image are listed separately so they do not count as false reports.
33.3 Criteria
Completeness (planted artefacts reported ÷ planted), false reports (artefacts reported that were not planted and are not OS defaults), time to result, and clarity of output.
Completeness and false reports measure accuracy, while time and clarity measure usability. Defining each criterion before running the tools prevents choosing measures that favour one tool. Excluding OS defaults from false reports is a fair rule, since a tool that reports a genuine system file is not wrong; the rule is written down in advance.
33.4 Confounders
Tool versions, settings, analyst experience and hardware. She fixes versions, uses default settings documented in advance, runs both on the same machine, and analyses outputs with a script rather than by eye where possible.
Each confounder could produce a difference that has nothing to do with tool quality. Running both tools on the same machine removes hardware differences; fixed versions prevent an update from changing results mid-study, which the worked example shows she handled. Using a script to compare outputs reduces the chance that her own expectations influence what she counts.
33.5 Analysis
Results are compared item by item, so every difference can be traced to a specific artefact.
Item-by-item comparison lets her explain every miss, not just count them. If a tool misses an artefact, she checks the raw image to confirm the artefact is present, then looks for the cause in the tool's documentation or settings. This produces explanations that tool users can act on, such as changing a default view. Each explanation is written beside the item in her results table, with the page of documentation or setting that confirms it, so a reader can check the cause independently.
33.6 Ethics
No real data; tools obtained from official sources; results reported fairly, including where her preferred tool performed worse.
Reporting fairly means stating results even when they disappoint her expectations. Obtaining tools from official sources avoids tampered copies and respects licences. Synthetic data mean no personal information is involved. Her pre-registered plan, shared with Dr Ferreira, is the record that her methods were fixed before results were known, which supports the claim that the comparison was fair and that results were not chosen after the fact.
33.7 Limitations
One image, one OS build, default settings; results describe this test, not general tool quality.
These limits are stated up front so readers do not generalise too far. A different OS build might change how prefetch entries are formatted, and different settings might change which files a tool shows. Her results tell examiners what to check on their own systems, not which tool is better overall. A reader who works on another build should repeat the planted-artefact test before relying on either tool for the same artefacts. The limitations paragraph says this in plain words and appears beside the results table rather than at the end.
Sample size and scope
With one image, Leah cannot estimate how often tools miss artefacts in general. She frames the study as a case study that finds specific blind spots, and suggests a larger set of images as later work. Framing the claim to match the design avoids overreach.
Worked example
Leah pre-registers her plan with Dr Ferreira: ground truth list, criteria, settings, analysis script. When a tool update arrives mid-study, she keeps the original version for consistency and notes the update as future work.
Synthetic case. The study is invented.
Errors and limits
Comparisons can be unfair if one tool needs a non-default option to parse an artefact; record such cases rather than declaring a "winner".
Exercises
- EX097 — Ground truth. Why fix ground truth before running tools? Rubric: 3 = avoids adjusting truth to outputs; enables fair scoring; 2 = partial; 1 = convenience; 0 = none.
- EX098 — Confounders. Name three confounders and their controls. Rubric: 3 = versions, settings, hardware/analyst with controls; 2 = two; 1 = one; 0 = none.
- EX099 — Mid-study update. How was the tool update handled? Rubric: 3 = kept original version; noted as future work; 2 = partial; 1 = switched; 0 = none.
Synthesis
Comparative studies need fixed ground truth, pre-set criteria, controlled confounders and fair reporting.
Continue studying — chapter 33 of 40
← Chapter 32 (draft) · Course chapter index · Chapter 34 (draft) →