◇ OpenCyber Academy
Doctoral research · Chapter 4 · English

Chapter 4 — Baseline replication and pilot

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 4 — Baseline replication and pilot

Lesson map

Lesson 04 published. Map, example, EX021.

flowchart TD
  N0["Learning outcomes"]
  N1["4.1 Why replicate first"]
  N2["Lab — carving pilot"]
  N3["4.2 Reading the result"]
  N4["4.3 Scoring by hash"]
  N5["4.4 Diagnosis"]
  N6["4.5 Feasibility decision"]
  N7["4.6 Baseline documentation"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX021"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["4.1 Why replicate first"]
  N2["Lab — carving pilot"]
  N3["4.2 Reading the result"]
  N4["4.3 Scoring by hash"]
  N5["4.4 Diagnosis"]
  N6["4.5 Feasibility decision"]
  N7["4.6 Baseline documentation"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX021"]

[Learning outcomes]
    |
    v
[4.1 Why replicate first]
    |
    v
[Lab — carving pilot]
    |
    v
[4.2 Reading the result]
    |
    v
[4.3 Scoring by hash]
    |
    v
[4.4 Diagnosis]
    |
    v
[4.5 Feasibility decision]
    |
    v
[4.6 Baseline documentation]
    |
    v
[Worked example] --> [Exercise EX021]

Learning outcomes

You will (1) replicate a baseline carving method on a small synthetic image; (2) score results against ground truth by hash; (3) diagnose why recoveries fail; (4) decide whether the pilot justifies the main study.

4.1 Why replicate first

A baseline that is wrongly implemented makes every later comparison meaningless. Ines first implements plain header/footer carving and runs it on an image whose truth she controls. The pilot image is small on purpose. With four files of known layout, Ines can predict by hand what a correct baseline should recover before she runs it. If the output differed from her prediction, either the carver or the scorer would be wrong, and she would fix that before generating hundreds of larger images for the main study.

Lab — carving pilot

"""Lab (ch 04): header/footer carving on a synthetic byte image, scored against ground truth (no real media)."""
import hashlib, random
random.seed(7)
H, F = b"\xff\xd8\xff", b"\xff\xd9"
def body(n): return bytes(random.randrange(0, 255) for _ in range(n)).replace(b"\xff", b"\x00")
truth = {name: H + body(n) + F for name, n in (("a", 300), ("b", 500), ("c", 200), ("d", 400))}
c1, c2 = truth["c"][:100], truth["c"][100:]
image = body(64) + truth["a"] + body(32) + truth["b"] + c1 + body(50) + c2 + body(20) + truth["d"][:250] + body(40)
carved, pos = [], 0
while (s := image.find(H, pos)) != -1:
    e = image.find(F, s)
    nxt = image.find(H, s + 1)
    if e == -1 or (nxt != -1 and nxt < e):
        carved.append(image[s:nxt if nxt != -1 else len(image)]); pos = s + 1; continue
    carved.append(image[s:e + 2]); pos = e + 2
th = {hashlib.sha256(v).hexdigest(): k for k, v in truth.items()}
intact = [th[hashlib.sha256(c).hexdigest()] for c in carved if hashlib.sha256(c).hexdigest() in th]
print("carved:", len(carved), "intact matches:", sorted(intact))
print("recall:", f"{len(intact)}/{len(truth)}", "precision:", f"{len(intact)}/{len(carved)}")

Run with python3 lab_carve.py. Verified output:

carved: 4 intact matches: ['a', 'b']
recall: 2/4 precision: 2/4

4.2 Reading the result

Four candidate files were carved. Two — a and b, both contiguous — match their ground-truth SHA-256 hashes exactly. File c was split into two fragments with 50 bytes of unrelated data between them; the carver ran from c's header to c's footer and captured the intruding bytes, so the hash does not match. File d was truncated: its footer is missing, so the carver ran to the end of the image. Recall and precision are both 2/4. Those two failures are the ones header/footer carving is expected to make, so the pilot confirms that both the generator and the scorer behave as designed. The 2/4 figure is not a study result; it shows only that the scoring counts each case correctly.

4.3 Scoring by hash

Hash equality is strict: one wrong byte fails. That is appropriate for "intact recovery". A secondary measure — byte overlap with truth — can credit partial recovery, and the thesis should report both. For file c in the pilot, byte overlap would show that most of the file was captured, with 50 intruding bytes, while hash equality marks it as lost. Reporting both prevents readers from confusing a damaged recovery with a complete miss. The scorer prints both measures for every file, so readers can see which hash misses still recovered most of the bytes and which recovered almost nothing.

4.4 Diagnosis

Each failure is explained by the manifest: fragmentation (c) and truncation (d). Being able to explain every miss is evidence the scoring works. Ines writes the diagnosis next to each failure in her analysis log, citing the manifest entry for the fragment boundaries of c and the missing footer of d. A miss that the manifest could not explain would mean a bug in the generator, the carver or the scorer, and she would stop to find it. Main-study misses are classified the same way, from the manifests.

4.5 Feasibility decision

The pilot passes: the generator and scorer behave as designed, and failures match known causes. The main study can scale to many images and conditions. Ines also uses the pilot to check effort: generating and scoring one small image takes little time, so she can estimate how long the main study's images will take before committing to the timeline. Prof. Achebe signs off the pilot in writing, and the sign-off is placed in the archive with the pilot log. Had either failure lacked a known cause, the main study would have waited.

4.6 Baseline documentation

The baseline's code, seed and outputs are archived so that later improvements are compared against exactly this version. Ines tags this version in her repository and refers to the tag in every later result table. If she later fixes a bug in the baseline, the fixed version gets a new tag, and any figure produced by it is labelled with that tag, so no reader confuses results from different versions of the carver. Rerunning the tagged baseline on the pilot image must again give 2/4.

Worked example

Ines's first scorer compared file sizes rather than hashes and reported c as recovered because its carved length was close. Switching to hashes exposed the error. She records the change in the analysis log.

Synthetic case. The image is invented teaching material.

Errors and limits

A pilot of four files proves mechanics, not performance.

Exercises

Synthesis

Replicate the baseline, score strictly by hash, explain every miss, and use the pilot to decide feasibility rather than performance.

Continue studying — chapter 04 of 08

← Chapter 03 (draft) · Course chapter index · Chapter 05 (draft) →