Chapter 40 — Verification laboratory II | Engineering Digital forensics and malware analysis

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 40 — Verification laboratory II

Lesson map

Lesson 40 published. Map, example, EX118.

flowchart TD
  N0["Learning outcomes"]
  N1["40.1 Campaign scope"]
  N2["40.2 Adversarial cases"]
  N3["40.3 Results"]
  N4["40.4 Triage"]
  N5["40.5 Retest"]
  N6["40.6 Release decision"]
  N7["40.7 Record"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX118"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["40.1 Campaign scope"]
  N2["40.2 Adversarial cases"]
  N3["40.3 Results"]
  N4["40.4 Triage"]
  N5["40.5 Retest"]
  N6["40.6 Release decision"]
  N7["40.7 Record"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX118"]

[Learning outcomes]
    |
    v
[40.1 Campaign scope]
    |
    v
[40.2 Adversarial cases]
    |
    v
[40.3 Results]
    |
    v
[40.4 Triage]
    |
    v
[40.5 Retest]
    |
    v
[40.6 Release decision]
    |
    v
[40.7 Record]
    |
    v
[Worked example] --> [Exercise EX118]

Learning outcomes

You will (1) run a full verification campaign on EvidenceDesk; (2) include adversarial test cases; (3) record and triage defects; (4) decide release readiness.

40.1 Campaign scope

All requirements FR1–FR5 and NFR1–NFR4, plus scenarios from the threat and failure analysis: edited entries, deleted entries, reordered entries, rebuilt chains, truncated images, wrong-copy selection and clock drift.

The scenarios come from earlier failure and threat analysis, so each one traces back to a risk someone identified on paper. Clock drift, for example, tests whether entries from a machine with a wrong clock are caught by the ordering checks. Truncated images test whether verification spots a file that is shorter than recorded. The full list is written before testing starts.

40.2 Adversarial cases

A classmate who did not build the system plays a careless or dishonest insider on synthetic data: editing files directly, deleting log lines, swapping images with similar names. Every attempt should be prevented or detected.

Builders unconsciously avoid the paths they know are weak. The classmate is given administrator access to the test server and a list of goals, such as removing evidence of a mistake, but no hints about the defences. They keep notes on every attempt, including ones they abandoned, which show where the system's design discouraged tampering.

40.3 Results

Of 46 test cases, 44 pass. Two fail: deleting the final custody entry goes undetected until the next anchor, and an export of a case with zero items crashes.

The first failure exposes a real gap: verification checks the chain up to the last anchor, so an entry added after that anchor and then removed leaves no trace. The second is an edge case nobody had tested because no real case has zero items, yet one could exist after an intake error. Both are logged as defects with the steps to reproduce them.

40.4 Triage

The deletion gap is severity high: a final entry could vanish for up to a day. Fix: anchor after each entry rather than daily. The crash is severity low but fixed anyway.

Severity reflects consequence: losing a custody entry undermines the record's completeness, which is the system's core purpose. Anchoring after every entry adds a small write each time, which performance tests show is negligible. The crash affects only an unusual case and loses no data, but leaving a known crash in an evidence tool would invite doubt, so it is fixed in the same release.

40.5 Retest

After fixes, the full campaign runs again; all 46 pass. Regression tests for both defects join the suite.

Running everything again, not just the two failed cases, guards against a fix that breaks something else. The regression tests encode each defect precisely: delete the final entry and expect detection; export an empty case and expect a valid, empty report. These now run on every future build, so the same faults cannot quietly return. The second run uses the same synthetic data and the same tester, so the only thing that changed is the code itself.

40.6 Release decision

Criteria: all Must requirements pass, no open high-severity defects, validation records complete. Met; release 1.5.0 is approved for the pilot.

Pilot approval is narrower than full use: EvidenceDesk will run on two benches for three months alongside the existing paper process. If the two disagree, the paper record governs and the discrepancy is investigated. The criteria and the decision are written down before release, so the choice to ship cannot be made on enthusiasm alone. A missed criterion would have meant fixing and retesting first, however close the deadline.

40.7 Record

The campaign report lists cases, results, defects, fixes and the release decision, signed by Priya and the reviewer.

The signed report is the document a court or accreditor would ask to see. Each test case has an identifier, expected result, actual result and date. Defects link to their fix commits and regression tests. The reviewer's signature shows that someone other than the builder checked the results before the release decision. Copies go to the quality manager and into the release folder, where they sit next to the validation report.

40.8 Lessons

Adversarial testers find problems builders miss.

The two defects were found by the classmate, not by Priya's own tests, and both lay on paths she had assumed nobody would take. She notes for future projects that an adversarial round should be planned early, not bolted on at the end, and that testers should include someone with no stake in the design looking good. Hugo adds that the same holds for lab procedures. Priya also records which of her own assumptions the classmate broke, so the threat analysis can be updated.

Test data management

Synthetic test cases live in a versioned folder with a manifest of hashes, so the campaign always runs on the same inputs. When a new case is added, the manifest is updated and reviewed like code, keeping old campaign results comparable.

Worked example

The tester deletes the last line of the custody log. Before the fix, the remaining chain verifies and matches the previous day's anchor, so nothing is flagged. After per-entry anchoring, the deletion is detected immediately because the latest anchor names a link that no longer exists.

Synthetic case. The campaign is invented.

Errors and limits

A passed campaign covers the cases imagined; new scenarios need new tests.

Exercises

EX118 — Two failures. What two tests failed? Rubric: 3 = final-entry deletion undetected; zero-item export crash; 2 = one; 1 = vague; 0 = none.
EX119 — Deletion fix. Why does per-entry anchoring fix the deletion gap? Rubric: 3 = latest anchor references removed link, detected at once; 2 = partial; 1 = vague; 0 = none.
EX120 — Release criteria. State the release criteria. Rubric: 3 = Musts pass, no open high defects, validation records complete; 2 = two; 1 = one; 0 = none.

Synthesis

Full campaigns with adversarial testers, honest triage and retests decide release readiness.

Continue studying — chapter 40 of 50

← Chapter 39 (draft) · Course chapter index · Chapter 41 (draft) →