Chapter 5 — Main study and independent validation | Doctoral research Digital forensics and malware analysis

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 05 published. Map, example, EX027.

flowchart TD
  N0["Learning outcomes"]
  N1["5.1 Design"]
  N2["Lab — recall with intervals"]
  N3["5.2 Reading the result"]
  N4["5.3 Why intervals"]
  N5["5.4 Independent validation"]
  N6["5.5 Interpretation"]
  N7["5.6 Deviations"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX027"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["5.1 Design"]
  N2["Lab — recall with intervals"]
  N3["5.2 Reading the result"]
  N4["5.3 Why intervals"]
  N5["5.4 Independent validation"]
  N6["5.5 Interpretation"]
  N7["5.6 Deviations"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX027"]

[Learning outcomes]
    |
    v
[5.1 Design]
    |
    v
[Lab — recall with intervals]
    |
    v
[5.2 Reading the result]
    |
    v
[5.3 Why intervals]
    |
    v
[5.4 Independent validation]
    |
    v
[5.5 Interpretation]
    |
    v
[5.6 Deviations]
    |
    v
[Worked example] --> [Exercise EX027]

Learning outcomes

You will (1) run the main study across controlled conditions; (2) report recall with interval estimates; (3) arrange independent validation; (4) interpret results without overreach.

5.1 Design

Three conditions — contiguous, 10% fragmented and 30% fragmented images — each with 50 target files generated from recorded seeds. The baseline carver runs on all three; outputs are scored by hash against manifests. With 50 files per condition, each recall figure rests on a count, such as 48 of 50 contiguous files recovered intact. The seeds are listed in the pre-registration, so nobody can later replace an awkward image with a more convenient one. The scorer runs unchanged from the pilot, and its tag is printed with each result table.

Lab — recall with intervals

"""Lab (ch 05): recall per synthetic test image with Wilson 95% intervals (stdlib only)."""
from math import sqrt
def wilson(k, n, z=1.96):
    p = k / n; d = 1 + z*z/n
    c = (p + z*z/(2*n)) / d; h = z*sqrt(p*(1-p)/n + z*z/(4*n*n)) / d
    return round(c - h, 3), round(c + h, 3)
runs = {"img-contig": (48, 50), "img-frag10": (41, 50), "img-frag30": (29, 50)}
for name, (k, n) in runs.items():
    print(name, f"recall={k/n:.2f}", "CI95=", wilson(k, n))

Run with python3 lab_wilson.py. Verified output:

img-contig recall=0.96 CI95= (0.865, 0.989)
img-frag10 recall=0.82 CI95= (0.692, 0.902)
img-frag30 recall=0.58 CI95= (0.442, 0.706)

5.2 Reading the result

Recall is 0.96 on contiguous images (95% Wilson interval 0.865–0.989), 0.82 at 10% fragmentation (0.692–0.902) and 0.58 at 30% (0.442–0.706). Recall falls as fragmentation rises, supporting H1. The intervals for contiguous and 30% conditions do not overlap; the 10% interval overlaps both neighbours, so the study cannot distinguish 10% from either extreme with confidence at this sample size. Ines words this carefully because the data supports a trend, not a precise curve. Her thesis reports all three intervals in one table and avoids saying that 10% fragmentation causes a specific drop. The point estimates are counts of 48, 41 and 29 recovered files out of 50.

5.3 Why intervals

A single number hides uncertainty. Laboratories citing an error rate need its range. The Wilson interval behaves better than the simple normal approximation when proportions are near 0 or 1. Ines's own figures show why. A contiguous recall of 0.96 from 50 files could reflect a true rate anywhere from about 0.87 to 0.99, and a laboratory using the carver should plan around the lower end. Reporting only 0.96 would invite readers to treat the method as almost perfect on unfragmented media. Ines computes the Wilson interval in her own script and prints it beside each count.

5.4 Independent validation

A second lab, given the generator, seeds and carver, reruns the study. ISO/IEC 27042 stresses repeatability (same lab, same method) and reproducibility (different lab or operator). Matching hashes of outputs show reproducibility. The second laboratory receives no results in advance, only the generator, seeds, carver, scorer and the README, so its run tests the archive as well as the method. Ines compares its output hashes and recall counts with hers condition by condition and records any difference with its cause, rather than averaging the two runs.

5.5 Interpretation

The finding applies to header/footer carving of JPEG-like files on these synthetic images. It does not state real-world error rates for any commercial tool. Ines repeats this boundary in the abstract, the results chapter and the conclusion, so that no single sentence can be quoted without its conditions. A laboratory that wants an error rate for its own carver on its own media must run its own validation; her generator gives it a way to do so. It cannot rank tools or predict recall on any real drive.

5.6 Deviations

One seed produced an image with an accidental extra header pattern in noise; it is kept and reported, not removed, because removing awkward data biases results. Ines also logs one change to scope. The reassembly method listed among the compared methods was not run in the main study; the pre-registered measures covered the baseline only, so the reassembly comparison moved to future work and the contribution statement was revised to match. The noisy image remains in the 30% condition, and its effect is described in the deviation log.

Sample size planning

Before the main run, Ines checks how wide intervals would be for different file counts. With 50 files, an observed recall near 0.8 gives an interval roughly 0.2 wide. Halving that width needs about four times as many files; she budgets that for the extended study.

Worked example

The validation lab's numbers match exactly for two conditions but differ by one file at 30%. Investigation shows they used a newer Python version whose random generator behaviour for a helper function differed. Ines pins versions in the archive and records the discrepancy.

Synthetic case. The figures are invented teaching material.

Errors and limits

Fifty files per condition gives wide intervals; more images would narrow them.

Exercises

EX027 — Interval reading. State recall and the 95% interval at 30% fragmentation.
Show marking criteria
Rubric: 3 = 0.58, 0.442–0.706; 2 = one; 1 = wrong; 0 = none.
EX028 — Overlap meaning. What does overlap of the 10% interval with both neighbours imply?
Show marking criteria
Rubric: 3 = cannot distinguish at this sample size; 2 = partial; 1 = significant; 0 = none.
EX029 — Wilson choice. Why use the Wilson interval?
Show marking criteria
Rubric: 3 = better near 0/1 than normal approximation; 2 = partial; 1 = no reason; 0 = none.
EX030 — Repeat vs reproduce. Distinguish repeatability from reproducibility.
Show marking criteria
Rubric: 3 = same lab/operator vs different; 2 = partial; 1 = same; 0 = none.
EX031 — Odd seed. Why keep the image with an accidental header pattern?
Show marking criteria
Rubric: 3 = removing awkward data biases; report it; 2 = partial; 1 = delete; 0 = none.
EX032 — Version drift. What caused the validation discrepancy and the fix?
Show marking criteria
Rubric: 3 = Python version difference; pin versions; 2 = one; 1 = vague; 0 = none.

Synthesis

Run controlled conditions, report intervals, seek independent reproduction and bound interpretation to what was tested.

Continue studying — chapter 05 of 08

← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →