Chapter 29 — Digital forensics and malware analysis: measurement | Engineering Digital forensics and malware analysis

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 29 — Digital forensics and malware analysis: measurement

Lesson map

Lesson 29 published. Map, example, EX085.

flowchart TD
  N0["Learning outcomes"]
  N1["29.1 What to measure"]
  N2["29.2 Accuracy test design"]
  N3["29.3 Completeness"]
  N4["29.4 Time"]
  N5["29.5 Protocol"]
  N6["29.6 Uncertainty"]
  N7["29.7 Baseline"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX085"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["29.1 What to measure"]
  N2["29.2 Accuracy test design"]
  N3["29.3 Completeness"]
  N4["29.4 Time"]
  N5["29.5 Protocol"]
  N6["29.6 Uncertainty"]
  N7["29.7 Baseline"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX085"]

[Learning outcomes]
    |
    v
[29.1 What to measure]
    |
    v
[29.2 Accuracy test design]
    |
    v
[29.3 Completeness]
    |
    v
[29.4 Time]
    |
    v
[29.5 Protocol]
    |
    v
[29.6 Uncertainty]
    |
    v
[29.7 Baseline]
    |
    v
[Worked example] --> [Exercise EX085]

Learning outcomes

You will (1) define measurements for EvidenceDesk; (2) measure accuracy, time and completeness; (3) design a measurement protocol; (4) report measurements with uncertainty.

29.1 What to measure

Accuracy: does verification correctly flag altered copies and pass unaltered ones? Completeness: are all required fields present? Time: how long do intake and verification take? Each measure ties to a requirement.

Accuracy maps to FR3, completeness to FR1 and FR2, and time to NFR4. Measuring things that map to no requirement can still be interesting, but those results go in a separate appendix so the main report stays tied to what was promised. Each measure also states what result would count as failure before any data is collected.

29.2 Accuracy test design

A synthetic set of 200 copies: 100 unaltered, 100 with single-byte changes at random offsets. Expected: 100 pass, 100 fail. Any other result is a defect.

Random offsets are chosen with a recorded seed, so the exact set can be regenerated. Priya also includes edge positions: the first byte, the last byte and bytes either side of a chunk boundary, since those are where reading bugs tend to hide. Expected outcomes are written into a file before the run, and the comparison is done by script rather than by eye.

29.3 Completeness

For 50 synthetic intakes by test users, count records missing required fields. The target is zero; the design should make it impossible.

Test users are given realistic pressure: some items arrive in batches, and one user is interrupted halfway through an intake. The count includes records that are complete but obviously wrong, such as a seizure time in the future, because validation should catch those too. A single missing field means a design flaw, not a user error. Each defect found this way gets a fix and a retest.

29.4 Time

Measure verification time per terabyte on Northfield's hardware, repeating runs to see variation. Report median and range.

Repeated runs pick up effects such as disk caching, background tasks and thermal throttling. Priya clears caches between runs where possible and records the conditions. She reports minutes per terabyte rather than raw times, so results from images of different sizes can be compared. Hardware details, including the storage type, go next to every figure. Runs that fail partway are kept in the record too.

29.5 Protocol

Written before running: inputs, procedure, expected results, pass criteria. Results recorded with tool version and hardware.

Writing the protocol first prevents quietly redefining success after seeing the data. If a run reveals a flaw in the protocol itself, such as a missing test case, the change is recorded as an amendment with its date and reason, and results from before and after the amendment are reported separately. The protocol is signed by Priya and Hugo before the first run, and the signed copy is stored with the results. A reader can then compare what was planned with what was done, line by line.

29.6 Uncertainty

Five timing runs of 31, 33, 32, 35 and 32 minutes per TB give a median of 32 and a range of 31–35. Reporting only "32 minutes" hides variation.

Five runs are enough to show variation but not to estimate it precisely, and the report says so. If NFR4 needs 1 TB in under four hours, even the slowest run, 35 minutes per terabyte, passes comfortably. Stating the margin against the requirement is more useful to managers than a long statistical discussion. More runs would be needed before any claim about typical speed on other hardware.

29.7 Baseline

Measure the old paper process too, so improvement is shown, not assumed.

Priya times paper intake for the same fifty synthetic items and counts missing fields on the resulting forms. Paper is faster per item but leaves gaps on many forms, while EvidenceDesk is a little slower and leaves none. Without the baseline, the slower intake time could look like a regression, whereas the comparison shows what the extra time buys. The baseline figures are kept for later comparisons. Paper timings exclude later data entry.

Who measures

Measurements run by the developer can be unconsciously favourable. Priya asks a classmate who did not build the system to execute the written protocol, and she records who ran each test. Where results differ from expectations, both review the logs before anyone changes code, so a fix is never made to match a number that was itself a mistake. Measurement records are stored beside the protocol, the tool version and the hardware description, so the quality manager can audit them later. Priya also notes which requirements have no measurement yet, so the gaps are visible rather than forgotten.

Worked example

Accuracy run: 100 unaltered passed, 100 altered failed. Completeness: 0 of 50 intakes incomplete. Paper baseline: 9 of 50 incomplete. Priya reports these counts with the protocol reference.

Synthetic case. The results are invented.

Errors and limits

Synthetic alterations test what they simulate; truncation and swapped files need their own cases.

Exercises

EX085 — Accuracy design. Describe the 200-copy accuracy test. Rubric: 3 = 100 unaltered, 100 single-byte changes; expected 100/100; 2 = partial; 1 = vague; 0 = none.
EX086 — Timing report. Report the five timing runs properly. Rubric: 3 = median 32, range 31–35 min/TB; 2 = one; 1 = mean only; 0 = none.
EX087 — Baseline value. Why measure the paper process? Rubric: 3 = shows improvement rather than assuming it (9/50 vs 0/50); 2 = partial; 1 = no reason; 0 = none.

Synthesis

Measure against requirements with written protocols, baselines and honest variation.

Continue studying — chapter 29 of 50

← Chapter 28 (draft) · Course chapter index · Chapter 30 (draft) →