Chapter 20 — Digital forensics and malware analysis: troubleshooting
Lesson map
Lesson 20 published. Map, example, EX058.
flowchart TD
N0["Learning outcomes"]
N1["20.1 Silent loss"]
N2["20.2 The symptom"]
N3["Lab — regex that skips lines"]
N4["20.3 Reading the result"]
N5["20.4 Prevention"]
N6["20.5 Root cause"]
N7["20.6 Reporting the fix"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX058"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["20.1 Silent loss"]
N2["20.2 The symptom"]
N3["Lab — regex that skips lines"]
N4["20.3 Reading the result"]
N5["20.4 Prevention"]
N6["20.5 Root cause"]
N7["20.6 Reporting the fix"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX058"]
[Learning outcomes]
|
v
[20.1 Silent loss]
|
v
[20.2 The symptom]
|
v
[Lab — regex that skips lines]
|
v
[20.3 Reading the result]
|
v
[20.4 Prevention]
|
v
[20.5 Root cause]
|
v
[20.6 Reporting the fix]
|
v
[Worked example] --> [Exercise EX058]
Learning outcomes
You will (1) troubleshoot forensic scripts and pipelines; (2) detect silent data loss; (3) fix regular-expression parsing errors; (4) prevent recurrence with tests.
20.1 Silent loss
The most dangerous bug in forensic tooling drops data without error. A parser that skips lines it cannot match produces a clean-looking but incomplete result.
A crash is visible and gets fixed; a skipped line looks like a quieter log. In an investigation that difference can change conclusions, because a count that is too low may move an hour below an alert threshold or remove the one event that linked two machines. Jonah's rule is that every parser must report how many lines it read, matched and skipped.
20.2 The symptom
Jonah's failed-logon count from FILESRV-02's text log is 38 for the attack hour; the server's own event viewer shows 41. Three events are missing.
The total of 41 quoted for that hour in the measurement chapter is the event viewer's figure. The coincidence with 38, the number of failures attributed to mfoster's desktop, is a trap: Jonah checks that the attribution was computed from the full set of events and not from his parser's incomplete output, and records the check in the case notes.
Lab — regex that skips lines
"""A timestamp regex that silently skips lines with milliseconds, and the fix."""
import re
lines = ["2026-05-04 08:01:12 fail admin", "2026-05-04 08:01:15.250 fail admin", "2026-05-04 08:01:19 fail admin"]
bad = re.compile(r"^(\d{4}-\d\d-\d\d \d\d:\d\d:\d\d) (\w+) (\w+)$")
good = re.compile(r"^(\d{4}-\d\d-\d\d \d\d:\d\d:\d\d(?:\.\d+)?) (\w+) (\w+)$")
print("bad regex matched:", sum(bool(bad.match(l)) for l in lines), "of", len(lines))
print("fixed regex matched:", sum(bool(good.match(l)) for l in lines), "of", len(lines))
Run with python3 lab_regex.py. Verified output:
bad regex matched: 2 of 3
fixed regex matched: 3 of 3
20.3 Reading the result
The original pattern requires times like 08:01:12 and silently rejects 08:01:15.250, which has milliseconds: it matches 2 of 3 synthetic lines. The fixed pattern makes the fractional part optional and matches 3 of 3. In the real log, three lines had milliseconds — the missing three events.
The lab reproduces the real fault with three lines so the cause is easy to see. A pattern written for one timestamp format is a hidden assumption, and the log changed format without notice. The fix is small, but finding it required noticing a difference of three between two counts and refusing to accept either until the difference was explained.
20.4 Prevention
The parser now counts unmatched lines and prints them; any non-zero count is investigated. A test with mixed timestamp formats is added.
Printing unmatched lines turns a silent fault into a visible one. The new test includes a line with milliseconds and one with a time zone suffix, so the next format change is caught on the test data rather than in a case. Jonah also adds the mixed-format test file to the repository, so anyone changing the parser runs it automatically.
20.5 Root cause
The log source switched to millisecond timestamps after a software update partway through the day.
The update changed the log format partway through the day, which explains why only some lines were affected. Jonah asks the server's owner for the update time and confirms it falls between the last line without milliseconds and the first line with them. The parser fault was real, but its trigger was a change nobody told the security team about. Change notices for log sources now go to the security team as well.
20.6 Reporting the fix
The case notes record the original count, the cause, the fix and the corrected count.
The note gives the version of the parser that produced each count, so a reader can see why an earlier figure differs from a later one. Correcting a number openly protects the case: an unexplained change from 38 to 41 between drafts would invite doubt about every other figure in the report. The corrected figure replaces the old one in every report that used it, and each report notes the change. Readers who saw the earlier draft are told directly.
20.7 General method
Compare outputs with an independent source; when counts differ, find out why before trusting either.
In this case the independent source was the server's event viewer. Other comparisons work too: the number of lines in a file against the number parsed, a tool's count against a second tool's count, or a collector's received count against the source's sent count. Whenever two counts differ, neither is trusted until the reason is found. Recording which comparison was made also tells a reviewer how the count was checked.
Pipeline monitoring
Jonah adds simple checks to the log pipeline: lines received per hour per source, compared with the previous week. A sudden drop means a source has stopped sending or a parser is failing; an alert goes to the IT team. Silent failures become visible failures, which can then be fixed before evidence is lost.
Worked example
After the fix, the count is 41, matching the event viewer. Jonah amends his earlier note: "Corrected 4 May 08:00–09:00 failed logons from 38 to 41; parser omitted lines with millisecond timestamps (fixed in script v1.3)."
Synthetic case. The log lines come from the lab script.
Errors and limits
Tests cover formats you anticipate; unmatched-line counts catch the ones you do not.
Exercises
- EX058 — Lab matches. How many lines did each regex match? Rubric: 3 = bad 2/3; fixed 3/3; 2 = one; 1 = wrong; 0 = none.
- EX059 — Root cause. Why did timestamps change format? Rubric: 3 = software update switched to milliseconds mid-day; 2 = partial; 1 = random; 0 = none.
- EX060 — Prevention. What prevents recurrence? Rubric: 3 = count/print unmatched lines; mixed-format test; 2 = one; 1 = vague; 0 = none.
Synthesis
Hunt silent data loss by comparing with independent sources, counting what parsers skip and testing formats.
Continue studying — chapter 20 of 30
← Chapter 19 (draft) · Course chapter index · Chapter 21 (draft) →