Chapter 4 — Scripting and data formats
Lesson map
Lesson 04 published. Map, example, EX013.
flowchart TD
N0["Learning outcomes"]
N1["4.1 Formats"]
N2["4.2 Authentication results"]
N3["Lab — reading authentication results"]
N4["4.3 Reading the result"]
N5["4.4 Passing is not trust"]
N6["4.5 Clear output"]
N7["4.6 Bad input"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX013"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["4.1 Formats"]
N2["4.2 Authentication results"]
N3["Lab — reading authentication results"]
N4["4.3 Reading the result"]
N5["4.4 Passing is not trust"]
N6["4.5 Clear output"]
N7["4.6 Bad input"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX013"]
[Learning outcomes]
|
v
[4.1 Formats]
|
v
[4.2 Authentication results]
|
v
[Lab — reading authentication results]
|
v
[4.3 Reading the result]
|
v
[4.4 Passing is not trust]
|
v
[4.5 Clear output]
|
v
[4.6 Bad input]
|
v
[Worked example] --> [Exercise EX013]
Learning outcomes
You will (1) parse simple key-value data with Python; (2) read email authentication results; (3) handle JSON, CSV and STIX at a basic level; (4) write scripts that report clearly.
4.1 Formats
Intelligence arrives as CSV lists, JSON (including STIX 2.1 bundles), plain email and log lines with key=value pairs. Python's standard library reads all of them: csv, json and simple string splitting.
Tomás reads each format with the matching module and prints a few records before trusting any value in a script. CSV lists from vendors may rename columns without warning, so he reads them by header name. STIX bundles are JSON, so the same json module handles both ESX bundles and ordinary JSON exports from college tools.
4.2 Authentication results
Mail gateways add an Authentication-Results header with SPF, DKIM and DMARC outcomes. DMARC passes only when the visible From domain aligns with an SPF or DKIM pass. A message failing DMARC that claims to come from the college deserves attention.
SPF checks the sending server and DKIM checks a signature, but DMARC also requires the domain to match the visible From address. That alignment rule is why a lookalike domain can pass SPF for itself and still fail DMARC. Tomás reads these results from Ashgrove's own gateway header, which the lab simulates with four synthetic messages.
Lab — reading authentication results
"""Lab (ch 04): parse synthetic Authentication-Results headers and flag messages failing DMARC."""
msgs = [("M1", "fees@ashgrove.example", "spf=pass dkim=pass dmarc=pass"),
("M2", "fees@ashgrove-payments.example", "spf=pass dkim=none dmarc=fail"),
("M3", "it-helpdesk@ashgrove.example", "spf=fail dkim=fail dmarc=fail"),
("M4", "library@partner-uni.example", "spf=pass dkim=pass dmarc=pass")]
for mid, sender, ar in msgs:
res = dict(kv.split("=") for kv in ar.split())
flag = "SUSPECT" if res["dmarc"] != "pass" else "ok"
print(mid, sender.split("@")[1], res, flag)
Run with python3 lab_headers.py. Verified output:
M1 ashgrove.example {'spf': 'pass', 'dkim': 'pass', 'dmarc': 'pass'} ok
M2 ashgrove-payments.example {'spf': 'pass', 'dkim': 'none', 'dmarc': 'fail'} SUSPECT
M3 ashgrove.example {'spf': 'fail', 'dkim': 'fail', 'dmarc': 'fail'} SUSPECT
M4 partner-uni.example {'spf': 'pass', 'dkim': 'pass', 'dmarc': 'pass'} ok
4.3 Reading the result
M1 and M4 pass all checks. M2 comes from ashgrove-payments.example, a lookalike; SPF passes for that domain but DMARC fails — flagged suspect. M3 claims the real ashgrove.example domain but fails everything — likely spoofed, flagged suspect.
M2 shows why one passing check is not enough: SPF passed only because the lookalike domain's owner published records for it. M3 is the more direct impersonation of the college, failing all three checks while claiming ashgrove.example as its domain. Both are flagged in the output, and the script prints the parsed results so the reasons for each verdict are visible.
4.4 Passing is not trust
M4 passes because partner-uni.example's records are correct; it shows the sender controls that domain, not that the message is safe. A compromised partner mailbox would also pass.
A passing result therefore tells Tomás where the message came from, not whether its contents are honest. If ESX warns that a partner university's mailboxes have been taken over, messages like M4 need the same suspicion as a failing one. He would then look at sender behaviour and content reports from staff, not at authentication results alone.
4.5 Clear output
Each line shows message, domain, parsed results and verdict. Anyone can check it without reading code.
In the lab output, each line carries the message identifier, the sender domain, the three parsed results and the verdict. The email team can read those lines and agree or disagree with each verdict without opening the script. Tomás pastes the output into tickets unchanged, so the reasoning behind each flag travels with it to whoever acts. If a verdict looks wrong, the email team can point to the exact line and parsed result in their reply.
4.6 Bad input
Malformed headers should produce "unparsed" with the raw text, not a crash.
A header that cannot be split into key=value pairs would raise an error in the lab's dict call and stop the whole run. Tomás's version catches that case, prints 'unparsed' with the raw text and carries on with the remaining messages. A person then reads the raw header, so an odd or forged result is examined rather than lost. The count of unparsed headers is printed at the end of each run, even when it is zero, so readers know the check happened.
4.7 STIX basics
A STIX indicator object holds a pattern, a valid_from time and an id. Tomás reads bundles with json.load and prints each object's type and id before using any value.
Printing the type and id first shows Tomás what the bundle contains before any pattern reaches a block list. A bundle may mix indicators with reports and relationships, and only indicator patterns are candidates for controls. The valid_from time tells him how old the indicator is, which matters for the expiry rules applied later.
Worked example
The finance office reports an invoice from ashgrove-payments.example. Tomás runs his parser on the saved header: DMARC fails, and the domain is a lookalike registered last week. He sends the sender domain to the email team for blocking and to ESX at TLP:GREEN without the staff recipient's name.
Synthetic case. The messages are invented.
Errors and limits
Headers added before the college gateway may be forged; trust only those added by Ashgrove's own gateway.
Exercises
- EX013 — Lab verdicts. Which lab messages were suspect and why? Rubric: 3 = M2 lookalike DMARC fail; M3 spoofed all fail; 2 = one; 1 = wrong; 0 = none.
- EX014 — Partner pass. What does M4's pass prove, and what does it not? Rubric: 3 = sender controls domain; not message safety/compromise possible; 2 = partial; 1 = safe; 0 = none.
- EX015 — Trusted headers. Which authentication headers should Tomás trust? Rubric: 3 = those added by Ashgrove's gateway; others forgeable; 2 = partial; 1 = all; 0 = none.
- EX016 — Malformed header. How should the script treat a malformed header? Rubric: 3 = report unparsed with raw text, no crash; 2 = partial; 1 = crash/skip; 0 = none.
Synthesis
Parse common formats with the standard library, read authentication results carefully and report clearly.
Continue studying — chapter 04 of 18
← Chapter 03 (draft) · Course chapter index · Chapter 05 (draft) →