Chapter 8 — Programming II
Lesson map
Lesson 08 published. Map, example, EX022.
flowchart TD
N0["Learning outcomes"]
N1["8.1 Parsing logs"]
N2["8.2 Functions"]
N3["8.3 Outputs"]
N4["8.4 Edge cases"]
N5["8.5 Do not modify inputs"]
N6["8.6 Readability"]
N7["8.7 Version control"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX022"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["8.1 Parsing logs"]
N2["8.2 Functions"]
N3["8.3 Outputs"]
N4["8.4 Edge cases"]
N5["8.5 Do not modify inputs"]
N6["8.6 Readability"]
N7["8.7 Version control"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX022"]
[Learning outcomes]
|
v
[8.1 Parsing logs]
|
v
[8.2 Functions]
|
v
[8.3 Outputs]
|
v
[8.4 Edge cases]
|
v
[8.5 Do not modify inputs]
|
v
[8.6 Readability]
|
v
[8.7 Version control]
|
v
[Worked example] --> [Exercise EX022]
Learning outcomes
You will (1) parse structured text logs; (2) use functions and error handling; (3) write outputs others can verify; (4) test parsers with edge cases.
8.1 Parsing logs
Logs are lines with fields: time, host, destination, method, path. Python's split() handles simple space-separated logs; the csv module handles quoted fields properly.
A simple split breaks as soon as a field contains a space, such as a user-agent string, so the parser must match the log's real format. Leah checks the proxy's documentation for its field order and quoting rules before writing code. She then tests the parser on a handful of real-looking lines whose correct fields she has written out by hand.
8.2 Functions
Leah writes parse_line(line) returning a dictionary, and load(path) that applies it to each line. Malformed lines are counted and reported, not silently dropped.
Splitting the work into two functions means parse_line can be tested on single lines, while load handles files. Counting malformed lines matters because silently dropping them could hide exactly the events an attacker caused, such as a request with an unusual path. The worked example shows the payoff: three malformed lines out of 2,418, each listed by number.
8.3 Outputs
Results are written to CSV with a header row, plus a small summary file recording input file name, its SHA-256, line count, malformed count and script version. Anyone can rerun the script and compare.
The input hash proves which log was processed. If someone later receives a different copy of the proxy log, comparing hashes shows at once whether it is the same file. Recording the script version ties the output to the code that produced it, so a bug found later can be traced to the reports it affected.
8.4 Edge cases
Empty lines, extra spaces, missing fields, non-ASCII characters in paths, and times that cross midnight. Each gets a test.
Times that cross midnight are a common trap: a log that records only times, with the date in the file name, will sort 00:01 before 23:59 unless the parser adds the date. Non-ASCII characters can break decoding if the file is read with the wrong encoding. Leah writes one test for each case, using a short invented line, so each failure points to one cause.
8.5 Do not modify inputs
The script opens logs read-only and writes outputs elsewhere.
Even a script that only reads can change a file's last-access time on some systems, so Leah works on copies whose hashes match the originals. Output goes to a separate folder named for the run. If a script ever wrote to its input, the hash recorded at collection would no longer match, and the evidence's integrity would be in question. Leah's script refuses to run if the output folder is the same as the input folder, a simple check that prevents an expensive mistake.
8.6 Readability
Names such as dest_host beat d; a short docstring states what each function returns.
Readable code helps reviewers, who must understand what a script did before trusting its results. A variable called dest_host can be checked against the log documentation, while d tells the reader nothing. Leah's docstrings state the input, the output and what happens to malformed lines, which matches the summary file the script writes and makes both easy to compare. Clear names also help Leah herself when she returns to the script months later for the capstone.
8.7 Version control
The script lives in a repository; the report cites the commit used.
Citing the commit means a reviewer can check out exactly the code that produced the findings, even if the script has changed since. When Leah fixed the missing-path crash, the fix became a new commit, and the report records which one was used. Earlier outputs remain traceable to the older version, so nobody confuses results from the two. If the fix had changed any counts, the report would say which figures were recomputed and with which commit.
8.8 Sharing
Scripts used for findings are kept with the case so reviewers can inspect them.
A reviewer who cannot inspect the script must trust its output blindly, which defeats the purpose of review. Keeping the script, its tests, the input hashes and the summary files together in the case folder lets another examiner rerun the analysis and get the same numbers. At Harrowgate, the case folder is stored on a restricted share with access logged by Grace Lin's office and retained with the case.
Character encodings
Logs may contain UTF-8 paths, and a parser that assumes ASCII can crash or garble them. Leah opens files with an explicit encoding and an error policy that records replaced characters, so unusual names stay visible instead of disappearing. Her tests include a synthetic path with accented characters.
Worked example
Leah's parser crashes on a proxy log line with a missing path field. She changes parse_line to return None for malformed lines, counts them, and the summary reports "2,418 lines parsed, 3 malformed (line numbers 77, 1,204, 2,001)". Dr Ferreira notes that listing line numbers lets a reviewer inspect exactly what was skipped.
Synthetic case. The log is invented.
Errors and limits
A parser encodes assumptions about format; if the format changes, results can be silently wrong — test on samples from each source.
Exercises
- EX022 — Malformed lines. How should malformed lines be handled? Rubric: 3 = counted, line numbers reported, not dropped silently; 2 = partial; 1 = crash/ignore; 0 = none.
- EX023 — Summary file. What does the summary file record? Rubric: 3 = input name, hash, line count, malformed count, script version; 2 = three; 1 = one; 0 = none.
- EX024 — Midnight case. Why test logs that cross midnight? Rubric: 3 = time-only fields can sort wrongly without dates; 2 = partial; 1 = vague reason; 0 = none.
Synthesis
Parsers that report what they skipped and record what they read make log analysis reviewable.
Continue studying — chapter 08 of 40
← Chapter 07 (draft) · Course chapter index · Chapter 09 (draft) →