Chapter 8 — Programming II

Lesson map

Lesson 08 published. Map, example, EX022.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["8.1 Collections"]
  N2["8.2 Parsing and exceptions"]
  N3["8.3 Normalization with provenance"]
  N4["8.4 Deduplication"]
  N5["8.5 Modularity and testing"]
  N6["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX022"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["8.1 Collections"]
  N2["8.2 Parsing and exceptions"]
  N3["8.3 Normalization with provenance"]
  N4["8.4 Deduplication"]
  N5["8.5 Modularity and testing"]
  N6["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX022"]

[Learning outcomes and connection]
    |
    v
[8.1 Collections]
    |
    v
[8.2 Parsing and exceptions]
    |
    v
[8.3 Normalization with provenance]
    |
    v
[8.4 Deduplication]
    |
    v
[8.5 Modularity and testing]
    |
    v
[Chapter synthesis]
    |
    v
[Worked example] --> [Exercise EX022]

Learning outcomes and connection

You will be able to (1) use lists, dictionaries and sets to group records; (2) parse JSON safely and handle malformed input with exceptions; (3) normalize timestamps and entity names while preserving the raw record; (4) deduplicate with an explicit key; and (5) structure code into testable functions with provenance. Chapter 3 validated flat records; here you handle nested, messy alert data. Chapter 9 moves from files to databases.

8.1 Collections

A list keeps order; a dictionary maps keys to values; a set holds unique items. For alerts, a dictionary keyed by a deduplication key groups duplicates; a set of entity names answers "which hosts appear?". Choose the structure that makes the question easy.

In Lab 08 the groups dictionary maps each deduplication key to a list of line numbers, so a group keeps every member. A set would answer "which hosts appear" but would lose how often and when, so choose it only when counts do not matter. Mixing the two carelessly is a common source of quietly wrong totals.

8.2 Parsing and exceptions

json.loads turns text into Python objects or raises json.JSONDecodeError. Malformed input is normal in incident work — truncated exports, mixed encodings. Catch the specific exception, record the line number and the raw text, and continue. Never use a bare except: that hides programming errors.

Lab 08 shows the pattern: line 6 lacks a closing brace, json.loads raises, and the loop records the line number with the error message before moving on. The final output therefore reports five parsed records and one malformed, rather than five records and silence. A rising count of malformed lines is itself a signal worth watching.

8.3 Normalization with provenance

Normalization converts different representations to one: host names to lower case, timestamps to UTC ISO 8601. Each normalized value should be traceable to its raw source. A simple design keeps three things per record: the raw text, the parsed/normalized fields, and a list of notes describing every transformation (for example "timestamp had no offset; assumed UTC per source documentation"). That list is the record's provenance.

In Lab 08 line 4 has no offset, so the program assumes UTC and writes that assumption into the record's notes. If the sensor actually logged local time at UTC−3, the event shifts three hours.

8.4 Deduplication

Two alerts are duplicates only under a stated rule: same rule ID, same entity, same event time within a tolerance. Deduplicate into groups, keeping all raw members, rather than deleting. Counting groups and members separately lets a reviewer check the rule.

Lab 08 shows why the rule must be stated: lines 1 to 3 describe one event within two seconds, yet the fixed five-second buckets split them into two groups. Because members are kept rather than deleted, the error is visible in the output and can be corrected. A deleted duplicate cannot be recovered to test a better rule later.

8.5 Modularity and testing

Split code into parse_line, normalize, dedup_key and report. Each can be tested with small inputs. Modularity also helps review: a colleague can check the timestamp logic without reading the reporting code.

In Lab 08 the timestamp logic lives in parse_ts, so a reviewer can test it with a single string such as an offset of minus three hours without building a full alert. Keep each function small enough that its tests fit on a screen; when a function needs many tests to cover its branches, it is usually doing two jobs. Name functions after what they return, not how they work.

8.6 Worked example and lab (synthetic): alert bundle

Save as lab08.py.

# Lab 08 - parse, normalize and deduplicate synthetic JSON alerts, keeping raw records (stdlib only)
import json
from datetime import datetime, timezone

LINES = [
    '{"rule": "R-PS1", "host": "WS-014", "ts": "2026-03-04T09:00:04Z"}',
    '{"rule": "R-PS1", "host": "ws-014", "ts": "2026-03-04T09:00:06+00:00"}',
    '{"rule": "R-PS1", "host": "WS-014", "ts": "2026-03-04T06:00:05-03:00"}',
    '{"rule": "R-NET4", "host": "ws-020", "ts": "2026-03-04 09:10:00"}',
    '{"rule": "R-NET4", "host": "ws-020", "ts": "04/03/2026 09:10"}',
    '{"rule": "R-PS1", "host": "WS-014", "ts": "2026-03-04T09:30:00Z"',
]

def parse_ts(s, notes):
    try:
        dt = datetime.fromisoformat(s.replace("Z", "+00:00"))
    except ValueError:
        notes.append("unparseable timestamp kept raw")
        return None
    if dt.tzinfo is None:
        notes.append("no offset; assumed UTC per sensor documentation")
        dt = dt.replace(tzinfo=timezone.utc)
    return dt.astimezone(timezone.utc)

records, errors = [], []
for n, line in enumerate(LINES, 1):
    try:
        obj = json.loads(line)
    except json.JSONDecodeError as e:
        errors.append((n, e.msg)); continue
    notes = []
    host = obj["host"].lower()
    if host != obj["host"]:
        notes.append("host lower-cased")
    records.append({"line": n, "raw": line, "rule": obj["rule"], "host": host,
                    "utc": parse_ts(obj["ts"], notes), "notes": notes})

def key(r, tol=5):
    if r["utc"] is None:
        return (r["rule"], r["host"], "raw:" + str(r["line"]))
    return (r["rule"], r["host"], int(r["utc"].timestamp()) // tol)

groups = {}
for r in records:
    groups.setdefault(key(r), []).append(r["line"])
print("parsed", len(records), "malformed", errors)
for k, members in groups.items():
    print(k[0], k[1], "lines", members)
for r in records:
    if r["notes"]:
        print("line", r["line"], r["notes"])

Expected output:

parsed 5 malformed [(6, "Expecting ',' delimiter")]
R-PS1 ws-014 lines [1]
R-PS1 ws-014 lines [2, 3]
R-NET4 ws-020 lines [4]
R-NET4 ws-020 lines [5]
line 1 ['host lower-cased']
line 3 ['host lower-cased']
line 4 ['no offset; assumed UTC per sensor documentation']
line 5 ['unparseable timestamp kept raw']

Interpretation: lines 1–3 describe the same event within two seconds (09:00:04, 09:00:06 and 06:00:05 at UTC−3, which is 09:00:05 UTC), yet the bucketed key splits them: 09:00:04 falls in the bucket that starts at 09:00:00, while 09:00:05 and 09:00:06 fall in the bucket that starts at 09:00:05. This is a known weakness of fixed time buckets: events near a boundary separate. A better rule compares each event with the previous one in sorted order (gap ≤ tolerance). Line 5 cannot be placed in time at all and is kept separately rather than guessed. Line 6 is reported, not silently dropped.

8.7 Common errors and limits

• Deleting duplicates instead of grouping them.
• Fixed buckets that split near-simultaneous events.
• Assuming naive timestamps are local time or UTC without documentation.

• Catching all exceptions and losing the error message.

• Normalizing host names without recording the change, so a reviewer cannot tell that WS-014 and ws-014 were merged.

• Treating the five-second tolerance as fact rather than a choice that must be justified for the sensor.
• Testing parsing on clean lines only, so the first truncated export breaks the pipeline during an incident. Each of these errors is cheap to prevent early and expensive to discover after a report has gone out.

Exercises — chapter 8

Attempt each exercise before opening the worked results. Every case, organization, person, address and number in an exercise is synthetic unless a cited public source is named.

EX022 — Fix the deduplication. Steps: replace the bucketed key with a sorted-gap rule (same rule and host, gap to previous ≤ 5 seconds) and show lines 1–3 now form one group. Rubric: 3 = correct code and output, None timestamps handled separately; 2 = works but drops unparseable; 1 = not run; 0 = none.

EX023 — Provenance note. Steps: write the provenance entry a reviewer would need for line 4, including the source of the UTC assumption and how to verify it. Rubric: 3 = names the assumption, the documentation or configuration to check, and the effect if wrong (a three-hour shift for a UTC−3 sensor); 2 = partial; 1 = absent; 0 = none.

EX024 — Malformed-input policy. Steps: write a five-line policy for handling malformed alerts in a response pipeline. Rubric: 3 = keep raw, count and report, alert on rising rates, no guessing, owner for fixes; 2 = three of these; 1 = "skip bad lines"; 0 = none.

Chapter synthesis

Parse defensively, normalize with a note for every transformation, group rather than delete duplicates under a stated rule, and keep raw records alongside the clean view so every conclusion can be traced.