Chapter 4 — Scripting and data formats
Lesson map
Lesson 04 published. Map, example, EX013.
flowchart TD
N0["Learning outcomes and connection"]
N1["4.1 A parser is a claim-making instrument"]
N2["4.2 Worked fixture: two accepted rows and three reje"]
N3["4.3 Normalize only with an audit trail"]
N4["4.4 A CSV record is not always a physical line"]
N5["4.5 Make the transformation independently reviewable"]
N6["Subject synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX013"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["4.1 A parser is a claim-making instrument"]
N2["4.2 Worked fixture: two accepted rows and three reje"]
N3["4.3 Normalize only with an audit trail"]
N4["4.4 A CSV record is not always a physical line"]
N5["4.5 Make the transformation independently reviewable"]
N6["Subject synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX013"]
[Learning outcomes and connection]
|
v
[4.1 A parser is a claim-making instrument]
|
v
[4.2 Worked fixture: two accepted rows and three reje]
|
v
[4.3 Normalize only with an audit trail]
|
v
[4.4 A CSV record is not always a physical line]
|
v
[4.5 Make the transformation independently reviewable]
|
v
[Subject synthesis]
|
v
[Worked example] --> [Exercise EX013]
Learning outcomes and connection
The learner will read a small CSV log bundle, state a schema before parsing, reject ambiguous rows without deleting them, and produce a deterministic, inspectable summary. Chapter 3 showed why a matching principal label is not an identity join. Chapter 4 asks a prior question: did the transformation preserve the source fields and report its own failures? Chapter 5 will attach authority and provenance to the resulting evidence package. RFC 4180 documents a common CSV format, while the Python CSV library handles quoting and dictionary rows; neither publication says that a parsed row is authentic incident evidence. All LLR rows here are original fictional teaching data.
4.1 A parser is a claim-making instrument
Before writing a script, name the question and the decision it may support. In this chapter the question is: “Which supplied rows are structurally fit for a later request-ID comparison?” The output is not a statement about the source system's completeness. A transformation may preserve the original order and still silently drop malformed rows, coerce a missing field into an empty string, or join two records with reused identifiers. Each behavior changes what an analyst might conclude. A good parser therefore has two outputs: accepted records and an explicit rejection report. The rejection report is part of the result, not a pile of errors to hide.
Define a minimum schema for the lab: event_id is nonblank and unique within this supplied bundle; event_utc is a UTC timestamp with a trailing Z; source is one of edge or api; request_id and kind are nonblank. A unique event ID in one file is still not a cross-system join key. That requires producer semantics and coverage. Decide whether a blank request ID is permitted before observing its effect on a favored hypothesis. Here the lab rejects it because the exercise is a request-ID comparison; in another workflow the row might remain useful in a separate, explicitly lower-resolution collection.
The raw input is a byte sequence. The parser's normalized object is a new artifact with its own creation time, version, code and rejected-row count. Do not overwrite the raw bytes. An operator should record the character encoding, line-ending and export mechanism; changing CRLF to LF or reserializing JSON changes the bytes even if the visible fields look similar. The short inline fixture below avoids file I/O, so it cannot demonstrate that a real collection preserved bytes or that its source clock was correct. It demonstrates deterministic logic on an already supplied string.
4.2 Worked fixture: two accepted rows and three rejections
This is branch L4, a teaching copy unrelated to the main case's verified evidence. Its E1/E2 labels and familiar request IDs are deliberately copied into a new synthetic CSV to test the parser. Do not promote these rows into the Chapter 2 or 3 case record. Line 1 is the header; lines 2–6 are data. The line number in the rejection report is a position in this literal string, not a source-system event number.
event_id,event_utc,source,request_id,kind
E1,2026-10-11T12:00:03Z,edge,r-201,http_503
E2,2026-10-11T12:00:03Z,api,r-202,room_R42_write
E2,2026-10-11T12:00:04Z,api,r-203,room_R40_write
E3,bad-time,edge,r-204,http_200
E4,2026-10-11T12:00:05Z,edge,,http_503
Run the following code with Python 3 on a local interpreter, or trace it by hand. It uses only the literal in-memory string. The code explicitly detects extra/missing columns, duplicate accepted IDs, invalid times and blank required fields. Its order of checks is part of the lab contract: the second E2 is classified as a duplicate before any other check. For a real log format, build tests for quoted delimiters, embedded newlines, encoding errors, oversized fields and version changes before using a parser operationally.
import csv
import io
from datetime import datetime
data = """event_id,event_utc,source,request_id,kind
E1,2026-10-11T12:00:03Z,edge,r-201,http_503
E2,2026-10-11T12:00:03Z,api,r-202,room_R42_write
E2,2026-10-11T12:00:04Z,api,r-203,room_R40_write
E3,bad-time,edge,r-204,http_200
E4,2026-10-11T12:00:05Z,edge,,http_503
"""
required = ["event_id", "event_utc", "source", "request_id", "kind"]
accepted, rejected, seen = [], [], set()
reader = csv.DictReader(io.StringIO(data))
assert reader.fieldnames == required
for line, row in enumerate(reader, start=2):
if set(row) != set(required) or any(value is None for value in row.values()):
rejected.append((line, "column_count"))
continue
if not row["event_id"] or not row["request_id"] or not row["kind"]:
rejected.append((line, "missing_required"))
continue
if row["event_id"] in seen:
rejected.append((line, "duplicate_id"))
continue
try:
if not row["event_utc"].endswith("Z"):
raise ValueError("not UTC Z")
datetime.fromisoformat(row["event_utc"].replace("Z", "+00:00"))
except ValueError:
rejected.append((line, "bad_time"))
continue
if row["source"] not in {"edge", "api"}:
rejected.append((line, "unknown_source"))
continue
seen.add(row["event_id"])
accepted.append(row)
print("accepted_ids:", [row["event_id"] for row in accepted])
print("rejected:", rejected)
The expected output is accepted_ids: ['E1', 'E2'] and rejected: [(4, 'duplicate_id'), (5, 'bad_time'), (6, 'missing_required')]. This is an exact local result for the fixture, not an assertion that LLR's real API emitted the second line. Record the interpreter version if reproducing the run. A variant with E4's request ID set to r-205 accepts E4 as a third row; that change is a new input, so keep both transcripts. Another variant that changes the second E2's event ID to E5 accepts it if all other fields pass. It does not make its R40 change factually approved.
4.3 Normalize only with an audit trail
For each accepted row, keep the raw line position, original text or byte offset, parser version, reason for acceptance and normalized fields. The lab returns a Python dictionary so the student can inspect fields; it does not serialize a production evidence store. If you later emit JSON, choose a stable encoding and ordering for reproducibility, and be alert to duplicate JSON member names: many decoders keep a later value without warning. A schema version should tell future readers whether a missing field was absent in the source or introduced by transformation. Preserve unknown fields in a separate quarantine or reject them with a reason; never silently throw them away because they do not fit a favored analysis.
The two accepted rows contain different request IDs. Even if E1 and E2 share a timestamp at one-second precision, the normalized table does not join them. To turn them into one transaction, the gateway and API custodians would need to validate an explicit ID mapping or a trusted parent-child relationship plus source coverage. Likewise a parser can say “no accepted API row for r-201 in this five-line fixture,” but it cannot say “the API never received r-201.” The latter would require a complete, validated source over the relevant interval and a known ID scheme. This is the same evidence boundary taught in Chapter 2, now enforced by software.
A technical review should probe negative controls. Change one field at a time: malformed time, duplicate ID, blank request ID, extra column, an accepted row whose source label is outside the allowed set. Predict the rejection reason and compare it with the actual result. A parser that crashes on one bad line can lose later valid lines; a parser that keeps everything without flags can contaminate a join. The failure policy must be chosen for the decision. Here invalid rows remain visible in a rejection list. They should not be “repaired” by guesswork; request a corrected export from its custodian.
4.4 A CSV record is not always a physical line
The L4 fixture is simple: one logical CSV record per physical line. That makes the enumerate-based ordinal in §4.2 coincide with the visible line number. A quoted comma or quoted newline breaks that coincidence. RFC 4180 describes these conventions, and the Python CSV library parses them according to its dialect. Splitting on commas or newline characters before parsing would corrupt the input. A rejection receipt must state whether its coordinate is a logical record number, a physical line number or a byte range.
Use this separate, fictional L4-Q variant; it is not an addition to the main LLR incident:
event_id,event_utc,source,request_id,kind
Q1,2026-10-11T12:05:00Z,edge,r-901,"note,review"
Q2,2026-10-11T12:05:01Z,api,r-902,"line one
line two"
Q1,2026-10-11T12:05:02Z,api,r-903,write
Predict §4.2's parser output before substituting the text into its in-memory data variable. With its existing schema, Q1 and Q2 pass: the comma and newline are each part of one quoted kind field. The last Q1 is a duplicate accepted event ID. The exact lists are accepted_ids: ['Q1', 'Q2'] and rejected: [(4, 'duplicate_id')]. Here 4 is the script's logical record ordinal: header 1, first Q1 2, Q2 3, repeated Q1 4. The repeated Q1 begins at physical line 5, because Q2 spans lines 3–4. Describing tuple 4 as “physical line 4” would misdirect a reviewer. To record both coordinates in a real approved parser, track the CSV reader's physical line counter around each next-record operation and retain a bounded raw excerpt or byte range.
Change only the header of a copy to repeat event_id and omit kind. The existing header assertion must reject that file before row interpretation. This is desirable for the declared schema: an unchecked dictionary with repeated header names could overwrite a value. Return to the original L4-Q before reporting its accepted/rejected transcript. The negative control shows why “a parser ran” is weaker than “the expected schema was validated.”
4.5 Make the transformation independently reviewable
For a real permitted export, retain untouched bytes and an acquisition receipt first. Then check encoding, header, dialect and version; parse with bounded resource use; apply field-level rules; and emit both accepted rows and an explicit rejected-row report. Each output needs its own object ID, source coordinates, input digest, parser version, run time and operator. If the source changes its schema, quarantine it until an owner explains the new fields. A script that silently adapts to an unknown column order can generate plausible but false joins.
The L4 check order has a subtle effect: an ID enters the seen set only after its row is accepted. A malformed first occurrence does not reserve the ID; a valid later occurrence might pass. That is a declared policy, not proof that the producer never reused an ID. If the decision question concerns all producer duplicates, build a separate inventory over accepted and rejected rows. Likewise, a locally unique event_id can collide across exports; uniqueness needs a source and export scope. A reviewer should record these choices beside any negative claim about a missing request.
Five-point review drill: check exact schema; quoted-field handling; logical and physical coordinates; duplicate policy; source provenance. A correct L4-Q review says two records accepted, one logical record rejected at ordinal 4, physical line 5 for the rejected record, and no authenticated source claim. It keeps the fixture separate from N01–N06. This drill feeds Chapter 5, where the analyst must account for permission and custody of both raw and derived artifacts.
Common errors and limits
- Splitting CSV on commas instead of using a parser. Quoted fields can contain commas or line breaks.
- Normalizing a time string without preserving the original spelling, offset and source clock information.
- Reusing an event ID from a different producer as if it were globally unique.
- Treating clean parse output as an authenticated, complete log export.
- Editing the main case record when running a clearly marked L4 counterfactual.
Exercises — subject 4
EX013 — Schema contract. Objective: make the transformation reviewable. Inputs: the L4 CSV and §4.1. Write a five-row schema table with type, requiredness, accepted values or format, rejection reason and a source-semantic question for each field. State why event_id and request_id have different roles. Evidence: Markdown table. Rubric: 2 = all five fields and distinct identifier scopes; 1 = validation rules without producer questions; 0 = treats event_id as a human or global transaction key.
EX014 — Parser transcript and variants. Objective: reproduce a deterministic result. Run §4.2 offline or trace it. Record both exact printed lines, then apply the E4 request-ID variant in a copy and explain the changed accepted/rejected sets. Keep the original transcript. Evidence: two transcripts and a two-sentence limitation. Rubric: 2 = exact line numbers and variant result, with no real-case inference; 1 = counts only; 0 = silently drops rejected rows.
EX015 — Cross-source join review. Objective: resist a false match. Inputs: accepted E1/E2, N04/N05 and A01. Produce a four-column claim table: proposed join, supplied support, missing producer semantics and allowed next request. Include an alternative in which the gateway maps r-201 to r-202, marked hypothetical. Evidence: table, not an identity attribution. Rubric: 2 = no join from time/label alone and a bounded mapping request; 1 = uncertainty named but no discriminator; 0 = claims E1/E2 prove the same request.
EX016 — Review a parser failure. Objective: identify transformation risk. A learner changes the script to discard the rejected list and to replace bad-time with 12:00:00Z. Explain two ways this can change a later timeline, propose a corrected output contract and state who can authorize a source correction. Evidence: short review note with one negative control. Rubric: 2 = preserves raw/rejection provenance and rejects invented timestamps; 1 = mentions bad data without a control; 0 = accepts the invented time as evidence.
Subject synthesis
A parser is part of the evidence chain. It can make source limits visible or erase them. L4 has two accepted and three rejected rows, yet none is a new main-case observation. Chapter 5 takes that separation into authorization, raw-byte preservation and collection receipts.
Continue studying — chapter 04 of 18
← Chapter 03 · Course chapter index · Chapter 05 →
Continue to the next authored lesson.