Chapter 4 — Programming I | Engineering Digital forensics and malware analysis

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 4 — Programming I

Lesson map

Lesson 04 published. Map, example, EX010.

flowchart TD
  N0["Learning outcomes"]
  N1["4.1 Why programming"]
  N2["4.2 Reading in chunks"]
  N3["4.3 hashlib"]
  N4["Lab — chunked hashing and verification"]
  N5["4.4 Reading the result"]
  N6["4.5 Good habits"]
  N7["4.6 Testing your script"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX010"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["4.1 Why programming"]
  N2["4.2 Reading in chunks"]
  N3["4.3 hashlib"]
  N4["Lab — chunked hashing and verification"]
  N5["4.4 Reading the result"]
  N6["4.5 Good habits"]
  N7["4.6 Testing your script"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX010"]

[Learning outcomes]
    |
    v
[4.1 Why programming]
    |
    v
[4.2 Reading in chunks]
    |
    v
[4.3 hashlib]
    |
    v
[Lab — chunked hashing and verification]
    |
    v
[4.4 Reading the result]
    |
    v
[4.5 Good habits]
    |
    v
[4.6 Testing your script]
    |
    v
[Worked example] --> [Exercise EX010]

Learning outcomes

You will (1) write simple Python with the standard library; (2) hash data in chunks; (3) verify a copy against an acquisition hash; (4) show that a one-bit change is detected.

4.1 Why programming

Examiners use validated tools, but engineers build and test the pipelines around them. Priya's first assignment from Hugo: write a script that verifies an evidence copy without loading it all into memory.

Loading a 500 GB image into memory is impossible on an ordinary workstation, and copying it first doubles both storage and time. The assignment therefore teaches streaming: read a piece, process it, discard it. The same pattern underlies indexing, carving and hash-set matching, so the first script is a template for much of the later work in the course.

4.2 Reading in chunks

Large images do not fit in memory. Reading 4 KiB or larger chunks and feeding each into a hash object gives the same digest as hashing the whole file at once.

Hash functions are designed so that feeding data in pieces gives exactly the same result as feeding it all at once, provided the pieces arrive in order and none are skipped. The lab below feeds its 95,000-byte file in 4,096-byte chunks: 23 full chunks and a final chunk of only 792 bytes. A loop that dropped that short last piece would still print a plausible-looking digest, which is why the lab compares the chunked result with a hash of the whole file.

4.3 hashlib

Python's hashlib module provides SHA-256. The update() method absorbs bytes; hexdigest() returns the result as hex text.

The object returned by hashlib.sha256() keeps internal state between update() calls, so one object per file is essential; reusing it would mix two files into one digest. Priya also prints the algorithm name from the object's name attribute next to each digest, so a report can never show a bare hex string whose algorithm the reader has to guess. Calling hexdigest() does not reset the object, so it is called once, after the last chunk.

Lab — chunked hashing and verification

"""Hash a synthetic evidence file in chunks and verify it (stdlib only)."""
import hashlib
data = b"SYNTHETIC-EVIDENCE-" * 5000
h = hashlib.sha256()
for i in range(0, len(data), 4096):
    h.update(data[i:i+4096])
acq = h.hexdigest()
print("bytes:", len(data))
print("acquisition sha256:", acq[:16] + "...")
print("verify copy:", "MATCH" if hashlib.sha256(data).hexdigest() == acq else "DIFFER")
bad = bytearray(data); bad[1000] ^= 1
print("verify altered copy:", "MATCH" if hashlib.sha256(bad).hexdigest() == acq else "DIFFER")

Run with python3 lab_hash.py. Verified output:

bytes: 95000
acquisition sha256: d260c00b9c859434...
verify copy: MATCH
verify altered copy: DIFFER

4.4 Reading the result

The 95,000-byte synthetic file is hashed in chunks at acquisition. Hashing the copy whole gives the same digest: MATCH. Flipping a single bit at offset 1000 produces a completely different digest: DIFFER. This sensitivity is why hash values are used to show integrity, as ISO/IEC 27037 describes for preserving digital evidence.

The flipped-bit test is worth repeating at different offsets: the first byte, the last byte and a byte straddling a chunk boundary. A chunking bug would most likely show up at a boundary, so a test that only changes offset 1000 could miss it. Priya adds all three cases to her test file and keeps the synthetic inputs alongside the script.

4.5 Good habits

Use clear names, print what was compared, never overwrite input files, and keep scripts under version control so the exact code used can be produced later.

Printing what was compared means the output names both files, both digests and the verdict, not just 'OK'. A reviewer reading the output months later should be able to tell which copy was checked against which original. Version control lets Priya tag the commit used on each case, so the script that produced a result can be checked out exactly, even after later improvements.

4.6 Testing your script

Before trusting the script on evidence, test it on inputs whose answers are known, including an empty file.

The empty file is a classic edge case: its SHA-256 is a well-known constant, and a script that crashes or prints nothing for it has a bug. Other useful tests are a file exactly one chunk long, a file one byte longer, and a file the script has no permission to read. Each test has an expected outcome written down before it runs.

Choosing chunk size

Chunk size affects speed, not the digest. Very small chunks waste time on function calls; very large ones use memory. Priya measures a few sizes on synthetic files and picks 1 MiB, documenting why. A reviewer can rerun the measurement and reach the same choice.

Worked example

Priya's first version opened the file in text mode, which can alter line endings on some systems and break hashes. Hugo points it out; she switches to binary mode and adds a test comparing her result with a second tool.

Synthetic case. The data are generated by the lab script.

Errors and limits

A matching hash shows the copy equals what was hashed at acquisition; it does not show the acquisition itself was complete.

Exercises

EX010 — Lab outcome. What did the lab report for the copy and the altered copy? Rubric: 3 = MATCH, DIFFER with reason; 2 = results only; 1 = wrong; 0 = none.
EX011 — Text mode bug. Why does opening evidence in text mode risk wrong hashes? Rubric: 3 = newline/encoding translation alters bytes; 2 = partial; 1 = slower; 0 = none.
EX012 — Hash limit. What does a matching hash not prove? Rubric: 3 = completeness of original acquisition; 2 = partial; 1 = proves all; 0 = none.

Synthesis

Chunked hashing in binary mode, tested on known inputs, is the basic tool of evidence integrity.

Continue studying — chapter 04 of 50

← Chapter 03 (draft) · Course chapter index · Chapter 05 (draft) →