Chapter 36 — Digital forensics and malware analysis: performance and scale
Lesson map
Lesson 36 published. Map, example, EX106.
flowchart TD
N0["Learning outcomes"]
N1["36.1 The scale problem"]
N2["36.2 Capacity arithmetic"]
N3["Lab — capacity estimate"]
N4["36.3 Reading the result"]
N5["36.4 Bottlenecks"]
N6["36.5 Hashing during imaging"]
N7["36.6 Scaling EvidenceDesk"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX106"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["36.1 The scale problem"]
N2["36.2 Capacity arithmetic"]
N3["Lab — capacity estimate"]
N4["36.3 Reading the result"]
N5["36.4 Bottlenecks"]
N6["36.5 Hashing during imaging"]
N7["36.6 Scaling EvidenceDesk"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX106"]
[Learning outcomes]
|
v
[36.1 The scale problem]
|
v
[36.2 Capacity arithmetic]
|
v
[Lab — capacity estimate]
|
v
[36.3 Reading the result]
|
v
[36.4 Bottlenecks]
|
v
[36.5 Hashing during imaging]
|
v
[36.6 Scaling EvidenceDesk]
|
v
[Worked example] --> [Exercise EX106]
Learning outcomes
You will (1) estimate imaging and verification capacity; (2) find bottlenecks; (3) plan scaling; (4) keep integrity while scaling.
36.1 The scale problem
Garfinkel (2010) highlighted growing data volumes as a central challenge. For Northfield, an eight-drive backlog totalling 20.5 TB must be imaged and verified with limited hardware.
Drive sizes have grown far faster than read speeds, so each new generation of disks takes longer to image than the last. A backlog is not only a delay to one case; waiting items take storage space, and suspects or victims wait for answers. Priya's first step is to turn the backlog into numbers: drive sizes, read rates and the number of imaging stations available.
36.2 Capacity arithmetic
At a sustained 150 MB/s, one terabyte (10^12 bytes) takes about 1.85 hours to read. Verification needs a second full read.
The figure comes from dividing 10^12 bytes by 150 × 10^6 bytes per second, giving about 6,667 seconds. Priya uses decimal terabytes because drive makers label capacity that way. Sustained speed is lower than a drive's advertised peak, so she measures it on a real seized drive of each type. The calculation ignores setup time, which she adds separately as about fifteen minutes per drive. Doubling the read for verification doubles the total.
Lab — capacity estimate
"""Imaging capacity estimate for a synthetic lab backlog."""
drives_tb = [1, 2, 2, 4, 0.5, 1, 8, 2]
rate_mb_s = 150
hours = [tb * 1e6 / rate_mb_s / 3600 for tb in drives_tb]
print(f"drives={len(drives_tb)} total_tb={sum(drives_tb)} hours={sum(hours):.1f}")
print(f"with verify pass: {2*sum(hours):.1f} h; split over 2 imagers: {sum(hours):.1f} h each")
Run with python3 lab_capacity.py. Verified output:
drives=8 total_tb=20.5 hours=38.0
with verify pass: 75.9 h; split over 2 imagers: 38.0 h each
36.3 Reading the result
Reading 20.5 TB once takes 38.0 hours at 150 MB/s; with a verification pass it is 75.9 hours on one imager. Split across two imagers, each spends about 38.0 hours. The 8 TB drive alone dominates: about 14.8 hours per read.
Splitting the backlog evenly is harder than the arithmetic suggests, because the 8 TB drive cannot be split between stations. Priya assigns it to one imager and fills the other with the smaller drives, so both finish at roughly the same time. The numbers also show that a third imager would help only if storage writes keep pace, which the next measurement checks.
36.4 Bottlenecks
The slowest component sets throughput: source drive, interface, write blocker, or destination storage. Measure each; upgrading the wrong one wastes money.
Priya times a copy from a fast source to null storage, then from a test file to the destination, isolating read and write speeds. At Northfield, the network share used as destination writes at only 110 MB/s, below the drives' read rate. Buying faster write blockers would have changed nothing. Moving the destination to locally attached storage brings throughput up to the source limit.
36.5 Hashing during imaging
Computing the digest while imaging avoids an extra read for the acquisition hash. Verification of the stored image still needs a separate read, which is what proves the stored copy is correct.
Hashing in the same pass costs little extra time because modern processors compute SHA-256 faster than most disks deliver data. The acquisition hash describes what was read from the source. The verification read describes what was written to storage. If the two match, the copy is faithful; skipping the second read would leave no proof that the write itself succeeded.
36.6 Scaling EvidenceDesk
Records grow slowly; the heavy load is storage and hashing. Priya's design hashes on the imaging station and lets EvidenceDesk verify asynchronously, queueing jobs and reporting completion.
Even a large lab adds only thousands of records a year, which a single server handles easily. Hashing a multi-terabyte image, however, can take hours, and blocking the interface while it runs would stall intake. The queue lets staff carry on while verification runs, and each job's start, finish and result are written to the custody log, so the record shows when the image was confirmed.
36.7 Prioritisation
Urgent cases jump the queue; the queue order and reasons are logged.
A case with a suspect in custody facing a charging deadline is urgent, while a historic fraud review usually is not. Reasons must be specific, such as 'custody deadline 14:00 Friday', so later reviewers can see the queue was not reordered for convenience. Hugo reviews the log monthly and asks about any case that waited unusually long, so lower-priority work is not starved indefinitely. Only the duty supervisor can move an item up, and each move is visible to every examiner in the queue view.
Energy and heat
Running several imagers continuously raises heat and power load in a small lab. Priya checks circuit capacity and cooling before recommending more hardware.
Worked example
Hugo asks whether a third imager would clear the backlog in a week. Priya computes: 75.9 hours ÷ 3 ≈ 25.3 hours each, well within a week if destination storage keeps up — which she checks next, finding it handles about 400 MB/s, enough for two imagers at full speed but not three.
Synthetic case. The backlog is generated by the lab script.
Errors and limits
Real drives with bad sectors run slower; estimates assume steady reads.
Exercises
- EX106 — Lab hours. Report single-read and verified hours for the backlog. Rubric: 3 = 38.0 h and 75.9 h; 2 = one; 1 = wrong; 0 = none.
- EX107 — Third imager. Why might a third imager not help fully? Rubric: 3 = destination storage ~400 MB/s limits to two at full speed; 2 = partial; 1 = it always helps; 0 = none.
- EX108 — Hash while imaging. What does hashing during imaging save, and what not? Rubric: 3 = saves extra read for acquisition hash; still need verify read of stored copy; 2 = partial; 1 = saves all; 0 = none.
Synthesis
Do the arithmetic, measure each component, and never trade away the verification read.
Continue studying — chapter 36 of 50
← Chapter 35 (draft) · Course chapter index · Chapter 37 (draft) →