◇ OpenCyber Academy
Technologist · Chapter 23 · English

Chapter 23 — Digital forensics and malware analysis: evaluation

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 23 — Digital forensics and malware analysis: evaluation

Lesson map

Lesson 23 published. Map, example, EX067.

flowchart TD
  N0["Learning outcomes"]
  N1["23.1 Criteria"]
  N2["23.2 Before"]
  N3["23.3 After"]
  N4["23.4 Remaining gaps"]
  N5["23.5 Method"]
  N6["23.6 Reporting"]
  N7["23.7 Independent check"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX067"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["23.1 Criteria"]
  N2["23.2 Before"]
  N3["23.3 After"]
  N4["23.4 Remaining gaps"]
  N5["23.5 Method"]
  N6["23.6 Reporting"]
  N7["23.7 Independent check"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX067"]

[Learning outcomes]
    |
    v
[23.1 Criteria]
    |
    v
[23.2 Before]
    |
    v
[23.3 After]
    |
    v
[23.4 Remaining gaps]
    |
    v
[23.5 Method]
    |
    v
[23.6 Reporting]
    |
    v
[23.7 Independent check]
    |
    v
[Worked example] --> [Exercise EX067]

Learning outcomes

You will (1) evaluate the city's forensic readiness after the incident; (2) use measurable criteria; (3) compare before and after; (4) report to management.

23.1 Criteria

Log coverage (percentage of key sources forwarded); retention against policy; time to preserve key hosts; kit availability and validation; staff trained; controls in place.

Each criterion is tied to a lesson from the incident: desktop logs were not forwarded, the export log nearly expired, memory capture took about two hours, and only one person knew the kit. Measuring those specific weaknesses shows whether the improvements addressed what actually went wrong, rather than producing a general score. Every criterion has a target, so the evaluation can say met or not met.

23.2 Before

Desktop logs forwarded: 0%. Application export log retention: 30 days. Time to memory capture in this incident: about 2 hours. Staff trained on kit: 1 of 8.

These figures come from the incident record and city records at the time: the collector's source list showed no desktops, the application configuration showed 30-day retention, the case notes gave the capture time, and training records listed one trained person. Each baseline figure is recorded with its source so the comparison can be checked later. The baseline is fixed and not revised later.

23.3 After

Key log sources forwarded: 95% (eight old desktops pending). Retention: 12 months. Exercise time to memory capture: 35 minutes. Staff trained: 8 of 8.

The improvements are large, but the figures are not all measured the same way. The capture time of 35 minutes comes from an exercise, while the earlier two hours came from a real incident, so the comparison flatters the new process. Jonah says so beside the figure rather than leaving the reader to assume like is compared with like. Each after figure carries its source and date beside it in the table.

23.4 Remaining gaps

USB activity not logged; payments provider 30-day logs; eight old desktops cannot forward logs and are scheduled for replacement.

Listing remaining gaps is as important as listing improvements. USB copying could still move data without central record; the payments provider's 30-day logs could still expire before discovery; eight desktops still keep logs only locally. Each gap has an owner and a planned action, so the evaluation also works as a plan. Gaps without funding are marked as such, so the committee sees what depends on its decision.

23.5 Method

Each figure comes from records: collector source lists, configuration exports, exercise logs, training records.

Using records rather than estimates means another person can reproduce each figure. Jonah keeps the exports he used, such as the collector source list on the evaluation date, in the evaluation folder with hashes. When the county reviewer questioned the forwarding percentage, the source list let Jonah split the figure by server and desktop within minutes. Figures that rest on judgement rather than records are labelled as estimates in the table.

23.6 Reporting

A two-page evaluation with a table, gaps and next steps goes to the IT manager and city risk committee.

The table shows each criterion with its before and after figures and source; the gaps follow with owners and dates; next steps close the report. Two pages is enough for the risk committee to see what improved and what remains, while the supporting records sit in an appendix for anyone who wants detail. The report is dated and versioned, and the next evaluation compares against it directly.

23.7 Independent check

A colleague from the county IT team reviews the evaluation.

A reviewer from outside the city team brings a fresh view and no stake in the result. The county colleague's question about the combined forwarding figure is the kind of challenge an internal author misses because the meaning seems obvious. Jonah records the review, the questions asked and the changes made in response. The reviewer's name and role appear in the report, along with any point where the two of them still disagree, so readers can judge for themselves.

Cost of improvements

The evaluation lists costs: collector storage, staff time for training and exercises, and replacement of eight old desktops. Jonah sets these beside the incident's cost in staff hours and the risk to residents' data, so the risk committee can judge value. He marks which costs are one-off and which recur each year, and which improvements are still unfunded.

Next review

Readiness is re-evaluated every six months using the same criteria, so trends are visible and slippage is caught early.

Audience

The risk committee is not technical, so each criterion is explained in one plain sentence beside its figure, with a note on why it matters for residents' data.

Worked example

The county reviewer questions the 95% figure: does it count servers and desktops together? Jonah separates them: servers 100% (20 of 20), desktops 94% (132 of 140; the eight old desktops are the gap). The report now shows both counts, avoiding a misleading combined figure of 152 of 160.

Synthetic case. The figures are invented.

Errors and limits

Exercise times are faster than real incidents; treat them as best case.

Exercises

Synthesis

Evaluate readiness with sourced measures, show gaps honestly and invite independent review.

Continue studying — chapter 23 of 30

← Chapter 22 (draft) · Course chapter index · Chapter 24 (draft) →