Chapter 9 — Digital forensics and malware analysis: leadership and assurance
Lesson map
Lesson 09 published. Map, example, EX041.
flowchart TD
N0["Learning outcomes"]
N1["9.1 Quality management"]
N2["9.2 Metrics with care"]
N3["Lab — turnaround summary"]
N4["9.3 Reading the result"]
N5["9.4 Acting on metrics"]
N6["9.5 Proficiency testing"]
N7["9.6 Peer review"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX041"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["9.1 Quality management"]
N2["9.2 Metrics with care"]
N3["Lab — turnaround summary"]
N4["9.3 Reading the result"]
N5["9.4 Acting on metrics"]
N6["9.5 Proficiency testing"]
N7["9.6 Peer review"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX041"]
[Learning outcomes]
|
v
[9.1 Quality management]
|
v
[9.2 Metrics with care]
|
v
[Lab — turnaround summary]
|
v
[9.3 Reading the result]
|
v
[9.4 Acting on metrics]
|
v
[9.5 Proficiency testing]
|
v
[9.6 Peer review]
|
v
[Worked example] --> [Exercise EX041]
Learning outcomes
You will (1) lead a forensic unit with quality assurance; (2) use metrics such as turnaround percentiles; (3) run proficiency and peer review programmes; (4) prepare for external audit.
9.1 Quality management
Marta's unit runs a quality system: documented procedures, validated methods, competence records, proficiency tests, peer review and corrective actions. ISO/IEC 27041 supports assurance that methods are fit for purpose; laboratory accreditation schemes assess the wider system. In practice each element is visible in CF-21: the validated endpoint agent, competence records for memory capture, the proficiency test on cloud audit timestamps and corrective actions after the forensic gate failure. Assessors look for evidence that the system works on real cases, not only that documents exist.
9.2 Metrics with care
Averages hide long delays. Percentiles show the tail that frustrates legal teams. For a forensic unit, the tail is where legal deadlines are missed and cases stall. An average pulled down by many quick cases can look acceptable while two cases wait more than a month. Marta reports median, 90th percentile and count over 21 days together, so the tail stays in view. Legal teams care about the case that waits six weeks, not the typical one. Reporting the tail lets Marta discuss those cases by name with the people waiting for them.
Lab — turnaround summary
"""Turnaround-time summary for synthetic forensic cases (days)."""
import statistics as st
tat = [4, 6, 5, 7, 30, 8, 5, 6, 9, 41, 7, 6, 5, 8, 12, 6, 7, 5, 9, 6]
q = st.quantiles(tat, n=10)
print(f"cases={len(tat)} median={st.median(tat)} p90={q[8]:.1f} max={max(tat)}")
print("over 21 days:", sum(t > 21 for t in tat))
Run with python3 lab_tat.py. Verified output:
cases=20 median=6.5 p90=28.2 max=41
over 21 days: 2
9.3 Reading the result
Twenty cases: median 6.5 days, 90th percentile 28.2 days, maximum 41. Two cases exceeded 21 days. The median looks healthy; the tail shows two cases waiting a month or more, both involving encrypted devices awaiting key recovery. The figures come from the lab's 20 synthetic cases, of which 18 finish within 12 days. The two slow cases are not slow because of examiner speed, so asking examiners to work faster would not shorten them. The diagnosis points instead to dependencies outside the unit, such as key recovery for encrypted devices.
9.4 Acting on metrics
Rather than pushing examiners to speed up, Marta separates cases blocked on external dependencies and tracks them as a distinct queue, with escalation dates. Tracking blocked cases separately keeps the main queue's metrics honest and gives each blocked case an owner and a date to escalate. For encrypted devices awaiting keys, escalation might mean asking counsel to press the device owner or the provider. The metric then measures what the unit can actually influence. Each blocked case is reviewed weekly until its dependency is cleared.
9.5 Proficiency testing
Twice a year, each examiner analyses a prepared synthetic case with known answers; results feed training plans. The prepared cases include traps based on real errors, such as the cloud audit log that stores UTC but displays local time. Answers are scored against known results, and an examiner who misses one gets targeted training and a retest. Results feed competence records and training plans, not individual performance ratings. The test answers are prepared by a senior examiner not taking that round of the test.
9.6 Peer review
Every report gets technical review (findings supported) and administrative review (completeness, format). For CF-21 the technical reviewer checks, for example, that the backup-server staging finding cites the NAT table and the 02:35 bulk read. The administrative reviewer checks that the report has scope, limits and an evidence list in the agreed format. Both reviews are recorded with reviewer names and dates before the report is released. Review comments and the examiner's responses stay in the case file for later audit. A report cannot be released while a technical comment remains unresolved.
9.7 Audit readiness
Records are organised so an auditor can trace any case from intake to report. For CF-21 an auditor should be able to follow WS-07 from the triage collection through the full image, the timestamp analysis and the reviewed finding, to the sentence in the final report. Each step has a record with a person and a date. Gaps in that trail would be a finding in themselves. Marta samples cases herself each quarter to test that the trail holds before an auditor does.
Developing people
Marta pairs junior examiners with seniors on live cases, rotates staff between incident and casework duties, and funds training linked to proficiency results. Competence records list each examiner's authorised methods, updated after training and assessment.
Worked example
A proficiency test shows two of six examiners misinterpreted a timestamp from a cloud audit log that records times in UTC while the console displays local time. Marta adds a training module and a checklist item, then retests: all six pass.
Synthetic case. The data are generated by the lab script.
Errors and limits
Metrics drive behaviour; pair speed with quality measures to avoid encouraging shortcuts.
Exercises
- EX041 — Lab percentiles. Report median, p90 and cases over 21 days. Rubric: 3 = 6.5, 28.2, 2; 2 = two; 1 = one; 0 = none.
- EX042 — Median vs tail. Why report p90 as well as median? Rubric: 3 = median hides long delays; tail matters to users; 2 = partial; 1 = no reason; 0 = none.
- EX043 — Blocked queue. What did Marta do about the tail? Rubric: 3 = separate externally blocked queue with escalation; 2 = partial; 1 = push speed; 0 = none.
- EX044 — Proficiency gap. What did the proficiency test reveal and how was it fixed? Rubric: 3 = UTC vs local display misread; training, checklist, retest; 2 = partial; 1 = vague; 0 = none.
- EX045 — Two reviews. Distinguish technical and administrative review. Rubric: 3 = findings supported vs completeness/format; 2 = partial; 1 = same; 0 = none.
Synthesis
Leadership means measuring the right things, testing competence and keeping records audit-ready.
Continue studying — chapter 09 of 12
← Chapter 08 (draft) · Course chapter index · Chapter 10 (draft) →