Chapter 38 — Digital forensics and malware analysis: validation | Engineering Digital forensics and malware analysis

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 38 published. Map, example, EX112.

flowchart TD
  N0["Learning outcomes"]
  N1["38.1 What is validated"]
  N2["38.2 Validation plan"]
  N3["38.3 Evidence gathered"]
  N4["38.4 Review"]
  N5["38.5 Validation report"]
  N6["38.6 Revalidation triggers"]
  N7["38.7 Using CFTT results"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX112"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["38.1 What is validated"]
  N2["38.2 Validation plan"]
  N3["38.3 Evidence gathered"]
  N4["38.4 Review"]
  N5["38.5 Validation report"]
  N6["38.6 Revalidation triggers"]
  N7["38.7 Using CFTT results"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX112"]

[Learning outcomes]
    |
    v
[38.1 What is validated]
    |
    v
[38.2 Validation plan]
    |
    v
[38.3 Evidence gathered]
    |
    v
[38.4 Review]
    |
    v
[38.5 Validation report]
    |
    v
[38.6 Revalidation triggers]
    |
    v
[38.7 Using CFTT results]
    |
    v
[Worked example] --> [Exercise EX112]

Learning outcomes

You will (1) plan validation of EvidenceDesk as part of a forensic method; (2) write a validation plan; (3) record evidence of fitness for purpose; (4) state limitations.

38.1 What is validated

EvidenceDesk does not analyse evidence, but it is part of the method by which Northfield preserves and verifies it. ISO/IEC 27041 provides guidance on assuring that methods are suitable and adequate, including defining requirements and testing against them.

Validation here answers a narrow question: does EvidenceDesk reliably record custody and confirm that stored images match their acquisition hashes? It says nothing about whether an examiner's interpretation of a file is correct. Keeping that scope explicit stops the validation report being quoted in court as if it covered more than it did. Priya writes the scope statement before any testing and has Hugo check it.

38.2 Validation plan

Scope; requirements; test cases with expected results; acceptance criteria; environment; who executes; how results are recorded and reviewed.

Writing the plan first prevents the common trap of deciding what counts as success after seeing the results. Acceptance criteria are numbers where possible, such as zero missed corruptions across the accuracy test. The environment part of the plan records hardware, operating system and EvidenceDesk version, so a later reader can tell whether the results apply to their setup. The plan is approved by the quality manager before execution begins.

38.3 Evidence gathered

Vector tests (algorithm), the 200-copy accuracy test (detection), the completeness test, integration end-to-end runs, restore tests and chain tamper tests. Each result is linked to a requirement.

Linking each result to a requirement produces a traceability table, with a requirement in each row and the tests supporting it alongside. A requirement with no test is visible immediately, as is a test that supports nothing. In Priya's table, NFR3 on append-only records has only the chain tamper tests behind it, so she adds a test that tries to overwrite an existing record through the normal interface and expects a refusal.

38.4 Review

An independent reviewer — Northfield's quality manager plus an external examiner — reviews the plan, results and limitations before approval.

The person who built a system tends to test what they expect to work. An external examiner brings habits from another lab and asks different questions; here, they ask how EvidenceDesk handles an image split across several files, which the plan had not covered. The reviewers record their comments, Priya responds to each in writing, and approval is withheld until every comment is closed.

38.5 Validation report

Summarises what was tested, results, deviations, limitations and the approved scope of use (e.g. "supported imager versions X and Y only").

Limitations are the part most often omitted and most often needed. The report states, for example, that only two imager versions were tested and that results on network storage were not measured. The approved scope of use turns those limitations into operating rules, so staff know not to use EvidenceDesk outside them. The report is versioned and stored with the release it validated.

38.6 Revalidation triggers

New imager versions, code changes affecting records, new storage platforms.

Each trigger is tied to what it could break. A new imager version can change the log format that ingest relies on. A code change to records or chain logic can alter how entries hash. A new storage platform can change read behaviour and speed. Revalidation need not repeat everything; the plan lists which tests each trigger requires, and a small change may need only a subset. Unplanned changes trigger a review first.

38.7 Using CFTT results

For write blockers and imagers, Northfield cites NIST CFTT reports where available and runs its own spot checks.

NIST's Computer Forensics Tool Testing programme publishes detailed test reports for specific tool versions, which saves each lab repeating the same expensive testing. A report covers only the version tested and the conditions used, so Northfield checks that its model and firmware match. Spot checks on the lab's own hardware, such as attempting a write through the blocker, confirm that the published result holds in practice.

Validation records retention

Validation plans, results and approvals are kept as long as any case processed under that version might be challenged, which can mean many years. They are stored with the release they cover and indexed by version, so a lab can answer quickly which validation applied to a particular case.

Worked example

The external examiner asks whether truncated images are detected. The plan had no such case. Priya adds a test with images truncated by one sector; EvidenceDesk flags all via hash and size mismatch. The report records the added case and its origin in review.

Synthetic case. The review is invented.

Errors and limits

Validation covers tested conditions; the approved scope must be respected in use.

Exercises

EX112 — Plan contents. List five parts of the validation plan.
Show marking criteria
Rubric: 3 = scope, requirements, cases, criteria, environment/executor/recording; 2 = three; 1 = one; 0 = none.
EX113 — Added case. Which test was added after review, and with what result?
Show marking criteria
Rubric: 3 = truncated images; all flagged; 2 = partial; 1 = vague; 0 = none.
EX114 — Revalidation. Name two revalidation triggers.
Show marking criteria
Rubric: 3 = new imager versions, record-affecting code changes (or storage); 2 = one; 1 = vague; 0 = none.

Synthesis

Validation links requirements, tests and independent review to an approved scope of use.

Continue studying — chapter 38 of 50

← Chapter 37 (draft) · Course chapter index · Chapter 39 (draft) →