Chapter 3 — Programming foundations
Lesson map
Lesson 03 published. Map, example, EX007.
flowchart TD
N0["Learning outcomes"]
N1["3.1 Why program"]
N2["3.2 Splitting lines"]
N3["3.3 Counting"]
N4["Lab — parsing event lines"]
N5["3.4 Reading the result"]
N6["3.5 Safe handling"]
N7["3.6 Testing"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX007"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["3.1 Why program"]
N2["3.2 Splitting lines"]
N3["3.3 Counting"]
N4["Lab — parsing event lines"]
N5["3.4 Reading the result"]
N6["3.5 Safe handling"]
N7["3.6 Testing"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX007"]
[Learning outcomes]
|
v
[3.1 Why program]
|
v
[3.2 Splitting lines]
|
v
[3.3 Counting]
|
v
[Lab — parsing event lines]
|
v
[3.4 Reading the result]
|
v
[3.5 Safe handling]
|
v
[3.6 Testing]
|
v
[Worked example] --> [Exercise EX007]
Learning outcomes
You will (1) write simple programs to parse log lines; (2) use lists, dictionaries and counters; (3) handle malformed input safely; (4) test a parser on known lines.
3.1 Why program
Logs arrive as text. A short program can split lines into fields, count events and skip broken lines faster and more reliably than reading by eye.
Clinic 2 alone produces thousands of firewall lines a day, and the server adds its own events, so reading by eye cannot keep up even for one small client. Mei writes Python because a script applies the same rule to every line and gives the same answer each time, which makes the result something Raj can check rather than a personal impression.
3.2 Splitting lines
Each event line has a time, host, event ID and key=value fields. Splitting on spaces gives the parts; a dictionary maps event IDs to names such as "logon failure".
In the lab each line is split on spaces, so the first part is the time, the second the host, the third the event ID and the rest the key=value fields. The dictionary names holds four IDs: 4624 logon success, 4625 logon failure, 4720 account created and 4732 added to local group. Any line whose third part is not one of those four IDs is treated as unparsed.
3.3 Counting
Python's Counter adds up how often each event type appears.
Each recognised event adds one to the counter under its readable name, so the script prints a count per event type without Mei preparing a list of types beforehand. The lab uses counts[names[parts[2]]] += 1, and because Counter starts every key at zero there is no need to check whether an event type has been seen before. Mei checks the printed counts against the input, because a counter that silently skips an event type would make every later report wrong.
Lab — parsing event lines
"""Parse synthetic Windows-style event lines into fields and count by event type."""
from collections import Counter
lines = """2026-05-04T01:12:03Z CLINIC2-SRV 4625 user=reception src=203.0.113.45
2026-05-04T01:12:05Z CLINIC2-SRV 4625 user=admin src=203.0.113.45
2026-05-04T01:12:07Z CLINIC2-SRV 4625 user=administrator src=203.0.113.45
2026-05-04T01:19:41Z CLINIC2-SRV 4624 user=reception src=203.0.113.45
2026-05-04T01:22:10Z CLINIC2-SRV 4720 user=reception target=support2
broken line without fields
2026-05-04T01:22:30Z CLINIC2-SRV 4732 user=reception target=support2 group=Administrators""".splitlines()
names = {"4624": "logon success", "4625": "logon failure", "4720": "account created", "4732": "added to local group"}
counts, bad = Counter(), 0
for ln in lines:
parts = ln.split()
if len(parts) < 4 or parts[2] not in names: bad += 1; continue
counts[names[parts[2]]] += 1
for k, v in counts.items(): print(f"{k:22} {v}")
print("unparsed lines:", bad)
Run with python3 lab_parse.py. Verified output:
logon failure 3
logon success 1
account created 1
added to local group 1
unparsed lines: 1
3.4 Reading the result
From seven lines, the parser counts three logon failures, one logon success, one account creation and one addition to a local group, and reports one unparsed line. The broken line was skipped and counted, not silently lost.
Mei checks the arithmetic: three failures, one success, one account creation and one group change make six parsed lines, and the one unparsed line brings the total to seven, matching the input. If those numbers did not add up, a line would have been counted twice or lost, and she would fix the parser before trusting any result from it.
3.5 Safe handling
Real logs contain truncated or odd lines. Programs must skip them, count them and continue.
In the lab, the guard if len(parts) < 4 or parts[2] not in names catches the line "broken line without fields" and adds one to bad instead of stopping with an error. Without that guard the script would either crash or mislabel the line, and on a real export one corrupt line would end the run before the remaining events were counted. The count of skipped lines is itself useful, because a sudden rise in unparsed lines usually means the export format changed upstream.
3.6 Testing
Mei runs the parser on these known lines first; the expected counts are written down before running, so a mismatch shows a bug.
For the lab lines Mei writes the expected result in advance: three failures, one success, one account created, one group change and one unparsed line. The run matches those written expectations exactly, line for line. Had the script printed two failures, she would know the parser was wrong before using it on real exports, where nobody knows the correct answer beforehand. The written expectation also helps Raj, who can review the test lines and the expected counts without running the script himself.
Keeping it readable
Mei names variables clearly, adds a one-line comment saying what the script does and stores it in Copperleaf's shared repository so Raj can review and other analysts can reuse it. A script only its author can read becomes a problem the day that person is absent.
Rerun on new data
The same script runs unchanged on next week's export.
Worked example
The parsed lines tell a story: three failed logons from 203.0.113.45, a success as "reception" at 01:19, then that account created "support2" and added it to Administrators at 01:22. Mei notices that a reception account should never create accounts and raises it with Raj.
Synthetic case. Lines come from the lab script.
Errors and limits
Simple splitting breaks if values contain spaces; real parsers handle quoting.
Exercises
- EX007 — Lab counts. Report the counts from the parser. Rubric: 3 = 3 failures, 1 success, 1 created, 1 group add, 1 unparsed; 2 = most; 1 = few; 0 = none.
- EX008 — Broken lines. How should a parser treat broken lines? Rubric: 3 = skip, count and continue; 2 = partial; 1 = crash; 0 = none.
- EX009 — Story in the lines. What sequence do the parsed lines show? Rubric: 3 = failures, success as reception, new account support2 added to admins; 2 = partial; 1 = vague; 0 = none.
Synthesis
Simple parsers turn raw lines into countable events and reveal sequences worth investigating.
Continue studying — chapter 03 of 30
← Chapter 02 (draft) · Course chapter index · Chapter 04 (draft) →