Chapter 8 — Programming II
Lesson map
Lesson 08 published. Map, example, EX022.
flowchart TD
N0["Learning outcomes"]
N1["8.1 Feeds as files"]
N2["Lab — validating a feed"]
N3["8.2 Reading the result"]
N4["8.3 Validation rules"]
N5["8.4 Never silently drop"]
N6["8.5 Idempotence"]
N7["8.6 Encoding and delimiters"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX022"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["8.1 Feeds as files"]
N2["Lab — validating a feed"]
N3["8.2 Reading the result"]
N4["8.3 Validation rules"]
N5["8.4 Never silently drop"]
N6["8.5 Idempotence"]
N7["8.6 Encoding and delimiters"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX022"]
[Learning outcomes]
|
v
[8.1 Feeds as files]
|
v
[Lab — validating a feed]
|
v
[8.2 Reading the result]
|
v
[8.3 Validation rules]
|
v
[8.4 Never silently drop]
|
v
[8.5 Idempotence]
|
v
[8.6 Encoding and delimiters]
|
v
[Worked example] --> [Exercise EX022]
Learning outcomes
You will (1) parse a CSV indicator feed with the standard library; (2) validate each row and record reasons for rejection; (3) report counts honestly; (4) design scripts that can be audited.
8.1 Feeds as files
HSSC distributes a daily CSV with columns for type, value, first-seen date and TLP. Spreadsheets mangle such files — turning long numbers into scientific notation or stripping leading zeros — so Amara processes them with Python's csv module instead.
She keeps the original file unchanged and writes processed output to a new file, so any question about the feed can be checked against what HSSC actually sent. Opening the original in a spreadsheet is avoided even for a quick look, because saving it by accident could change values without anyone noticing.
Lab — validating a feed
"""Lab 1 (ch 08): parse a synthetic CSV feed with the csv module and count accepted/rejected rows."""
import csv, io
data = """type,value,first_seen,tlp
domain,portal-login.example,2026-09-01,TLP:GREEN
ipv4,203.0.113.50,2026-09-02,TLP:AMBER
domain,,2026-09-02,TLP:GREEN
sha256,abc123,2026-09-03,TLP:GREEN
url,http://files.example/x,2026-09-04,TLP:CLEAR
"""
ok, bad = [], []
for row in csv.DictReader(io.StringIO(data)):
if not row["value"]:
bad.append(("empty value", row)); continue
if row["type"] == "sha256" and len(row["value"]) != 64:
bad.append(("sha256 length", row)); continue
ok.append(row)
print("accepted", len(ok), "rejected", len(bad))
for reason, row in bad:
print(" reject:", reason, row["type"], row["first_seen"])
Run with python3 lab_csvfeed.py. Verified output:
accepted 3 rejected 2
reject: empty value domain 2026-09-02
reject: sha256 length sha256 2026-09-03
8.2 Reading the result
Three rows pass; two fail. One has an empty domain value; another claims to be a SHA-256 hash but has the wrong length. Each rejection is printed with its date and reason, so Amara can send precise feedback to HSSC rather than "your feed is broken".
The feedback goes to HSSC's contact with the row numbers, so the mistakes can be fixed at the source for every member. She also checks whether the same faults appeared in earlier files. None had.
8.3 Validation rules
Rules check presence (no empty value), shape (hash length, valid address), allowed vocabulary (TLP labels from FIRST TLP 2.0: CLEAR, GREEN, AMBER, AMBER+STRICT, RED) and date format. Rules should be few, explicit and tested.
Amara writes one test for each rule using a small file of good and bad rows, so changing a rule shows immediately which cases it affects. She keeps the list of rules in the script's documentation and shares it with HSSC, so the community knows what HCHT checks and can catch problems before the feed is sent. New rules are added only after a real problem shows they are needed.
8.4 Never silently drop
A script that skips bad rows without saying so makes counts untrustworthy. Always report accepted and rejected totals.
Amara's import prints both totals and writes rejected rows to a file with the reason for each. If the totals do not add up to the number of rows in the feed, the script stops with an error. A silent drop once hid a column change in a feed for a week, until a missing domain was noticed by chance, which is why the rule is now strict.
8.5 Idempotence
Running the import twice should not create duplicates. Use the value plus type as a key and update metadata rather than adding new rows.
Amara tests this by importing the same feed twice and checking that the row count does not change, while the last-seen date updates. Idempotence matters because imports are rerun after failures, and duplicates would inflate counts in reports and make one indicator appear to come from two sources, raising confidence that the evidence does not support.
8.6 Encoding and delimiters
Feeds arrive with different encodings and separators. Open files with an explicit encoding, sniff or declare the delimiter, and strip invisible characters such as byte-order marks from the first header. Many baffling "missing column" errors are just a hidden character in the first field name.
Amara adds both steps to her import function. The import prints the detected encoding and delimiter, so a change in how HSSC produces the file is noticed at once instead of appearing later as a puzzling error. Amara keeps a small set of sample files with different encodings as tests.
Worked example
The next day the feed contains a row with TLP "AMBER STRICT" (space instead of plus). Amara's script rejects it with reason "unknown TLP label". She asks HSSC to correct it rather than guessing, because treating it as AMBER would loosen a restriction.
Synthetic case. The feed rows are invented.
Errors and limits
Valid format is not truth. A well-formed indicator can be wrong or stale; validation is only the first gate.
Exercises
- EX022 — Lab rejections. Which two rows failed in the lab, and why? Rubric: 3 = empty domain (09-02); bad sha256 length (09-03); 2 = one; 1 = counts only; 0 = none.
- EX023 — TLP typo. Why must "AMBER STRICT" not be mapped automatically to AMBER? Rubric: 3 = would loosen restriction; ask sender; 2 = partial; 1 = harmless; 0 = none.
- EX024 — Re-run safety. How should the importer avoid duplicates when run twice? Rubric: 3 = key on type+value; update not insert; 2 = dedupe vaguely; 1 = delete all first; 0 = none.
Synthesis
Parse with code, validate explicitly, report rejections and keep imports idempotent.
Continue studying — chapter 08 of 40
← Chapter 07 (draft) · Course chapter index · Chapter 09 (draft) →