Chapter 12 — Laboratory: ingest and preserve synthetic incident records
Lesson map
Lesson 12 published. Map, example, EX045.
flowchart TD
N0["Learning outcomes and connection"]
N1["12.1 Freeze the intake contract"]
N2["12.2 Offline parser and rejection report"]
N3["12.3 Manifest, normalized view and query boundary"]
N4["12.4 Reviewer walkthrough and failure modes"]
N5["Subject synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX045"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["12.1 Freeze the intake contract"]
N2["12.2 Offline parser and rejection report"]
N3["12.3 Manifest, normalized view and query boundary"]
N4["12.4 Reviewer walkthrough and failure modes"]
N5["Subject synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX045"]
[Learning outcomes and connection]
|
v
[12.1 Freeze the intake contract]
|
v
[12.2 Offline parser and rejection report]
|
v
[12.3 Manifest, normalized view and query boundary]
|
v
[12.4 Reviewer walkthrough and failure modes]
|
v
[Subject synthesis]
|
v
[Worked example] --> [Exercise EX045]
Learning outcomes and connection
The learner will ingest a small multi-source bundle without overwriting originals, reject malformed and duplicate rows visibly, create a manifest and distinguish a normalized view from a source-authenticated conclusion. Chapter 11 separated raw time, correction and copy integrity. Chapter 12 combines that discipline with Chapter 4's parser contract and Chapter 9's scope and custody fields. Chapter 13 will use the accepted records—and the rejected-row gap—to explain two conflicting alerts. NIST SP 800-92 addresses log-management processes at a high level; the following schema, parser and case are original workshop designs, not instructions attributed to that 2006 guide.
12.1 Freeze the intake contract
Workshop K12 supplies three literal CSV strings called idp, host and app. They are synthetic bytes in this book. AUTH-K12 allows offline inspection only; no tenant query, production log export or retention change is authorized. For each source, the manifest should record producer claim, supplied-by, collection time, object ID, byte count, digest, encoding, newline convention, access scope, time basis and parser version. In this classroom exercise, “supplied by” is the exercise publisher, and actual producer authenticity is unknown. Record that gap instead of inventing a custodian signature.
Each CSV uses columns event_id, event_utc, principal, session, request, kind. An event ID need only be unique within its source bundle. The event time must parse as a timezone-aware UTC value ending in Z for this particular contract. IdP rows require a session; app rows require a request; host rows need neither. Principal and kind must be nonblank for all. A blank session in an app row is permitted by schema but weakens session correlation. One row can be structurally valid and still be irrelevant, false or misinterpreted. Producers may reuse an event ID in different namespaces, so the accepted key is (source, event_id), not event_id alone.
The supplied packet says IdP I1 accepted session s-7 for svc-book at 11:59:59; IdP I2 denied different session s-8 for the same principal at 12:00:01; host H1 reports runner boot b7 at 12:00:02; app A1 reports request r-601 with session s-7 as queued at 12:00:03. “Queued” is neither DB commit nor patron-visible correctness. App A2 has a malformed time; a later A1 is a duplicate ID within the app bundle. Those bad rows must remain visible as rejection evidence, not quietly disappear. The app sample may also be incomplete even if all its rows parse.
12.2 Offline parser and rejection report
Run this self-contained Python 3 program on the literal strings. It does no file or network I/O. It preserves each raw string in the bundle dictionary while deriving accepted keys and explicit rejection tuples. It is deliberately small: a production parser would also need tested encoding, quoted-newline, size, header-version and export-provenance behavior. Python's CSV documentation describes the reader used; it does not authenticate any record.
import csv
import io
from datetime import datetime
bundle = {
"idp": """event_id,event_utc,principal,session,request,kind
I1,2026-10-11T11:59:59Z,svc-book,s-7,,success
I2,2026-10-11T12:00:01Z,svc-book,s-8,,denied
""",
"host": """event_id,event_utc,principal,session,request,kind
H1,2026-10-11T12:00:02Z,svc-book,,,boot_b7
""",
"app": """event_id,event_utc,principal,session,request,kind
A1,2026-10-11T12:00:03Z,svc-book,s-7,r-601,queued
A2,not-a-time,svc-book,s-9,r-602,queued
A1,2026-10-11T12:00:05Z,svc-book,s-9,r-603,queued
""",
}
header = ["event_id", "event_utc", "principal", "session", "request", "kind"]
accepted, rejected = [], []
for source, raw in bundle.items():
reader = csv.DictReader(io.StringIO(raw))
if reader.fieldnames != header:
raise ValueError("unexpected header for " + source)
seen = set()
for line, row in enumerate(reader, start=2):
if set(row) != set(header) or any(v is None for v in row.values()):
rejected.append((source, line, "columns"))
continue
if not row["event_id"] or not row["principal"] or not row["kind"]:
rejected.append((source, line, "missing_core"))
continue
if source == "idp" and not row["session"]:
rejected.append((source, line, "missing_session"))
continue
if source == "app" and not row["request"]:
rejected.append((source, line, "missing_request"))
continue
if row["event_id"] in seen:
rejected.append((source, line, "duplicate_id"))
continue
try:
if not row["event_utc"].endswith("Z"):
raise ValueError("not UTC Z")
datetime.fromisoformat(row["event_utc"].replace("Z", "+00:00"))
except ValueError:
rejected.append((source, line, "bad_time"))
continue
seen.add(row["event_id"])
accepted.append(source + ":" + row["event_id"])
print("accepted:", accepted)
print("rejected:", rejected)
The expected accepted list is idp:I1, idp:I2, host:H1, app:A1. The expected rejections are app line 3 bad_time and app line 4 duplicate_id. Input line numbers are positions in the supplied strings. The script does not overwrite those strings, yet in a real intake process merely keeping a Python variable is not a durable original or a custody record. A controlled original copy, immutable/versioned storage appropriate to the environment and access logging would need an approved design.
12.3 Manifest, normalized view and query boundary
Construct a three-row manifest: K12-idp-raw, K12-host-raw, K12-app-raw. Each row should name the exact supplied bytes, claimed producer, exercise publisher, AUTH-K12, UTF-8 text, LF line endings, in-memory method, byte count, digest and copy/access status. Add a fourth row for K12-derived-v1, linked to all three raw object IDs and a code/version identifier; record accepted/rejected counts and creation time. The raw app object contains three data rows even though the derived view accepts only one. A report that lists just one app row without the rejection count hides an important source-quality problem.
A normalized event table can retain source name, raw line, original event time, parsed UTC, principal, session, request and kind. It should not replace raw producer time with a Chapter 11 correction: K11 is a separate timing variant, and K12 has no measured offsets. No app row proves a database write. Literal s-7 matches between I1 and A1, but a trustworthy cross-producer join requires both custodians to validate the session namespace, export provenance and source coverage. A parser's acceptance answers “does this row satisfy this schema?” rather than “did the stated event occur?” If A2's malformed row is later corrected by its source custodian, create K12-derived-v2 and preserve the v1 rejection, reason and corrected raw object separately.
Design a coverage test before using missing r-602 as evidence. Ask the app owner for a known control event in the relevant route/window, a query-run time, ingestion lag, filters, producer version, dropped-event counters and a statement about which writes emit records. If the known control is absent, the negative r-602 result is weak. If present, the interval/path confidence improves but does not become universal completeness. Do not synthesize a “fix” for A2's time to make it fit a desired story.
12.4 Reviewer walkthrough and failure modes
At review, one learner plays collector, another reviewer. The collector presents the three immutable classroom strings, receipt fields, parser version, exact output, rejected rows and derived view. The reviewer asks whether the app A1 queue status is mistaken for a commit, whether I2 denial is improperly joined to A1, and whether the original objects can be distinguished from a re-export. A strong submission can answer “unknown” with the source needed to resolve it. It identifies that line 4 repeats A1 within app, while I1 and H1 are allowed as source-specific IDs. It also states that a future reader must know which row IDs belong to which version.
If an intake tool crashes after the first bad row, later records may be lost. If it silently coerces a time, the later alert rules may invent order. If it deduplicates only on event_id globally, it may drop legitimate rows from another producer. If it deduplicates on the entire normalized row, it might keep two conflicting rows with the same source event ID. Review the failure policy against the decision, not merely the code style. Chapter 13 treats the rejection report and missing-telemetry possibilities as first-class evidence limits.
Common errors and limits
- Calling accepted rows an authenticated source or complete incident record.
- Failing to version and retain the raw bundle and the rejection report.
- Using a global event-ID namespace across unrelated producers.
- Treating queue acceptance as a reservation commit.
- Filling a malformed time by guesswork or importing K11's offsets into K12.
Exercises — subject 12
EX045 — Intake manifest. Objective: make originals reviewable. Draft the three raw-object manifest rows plus K12-derived-v1, including source claim, supplier, authority, method, encoding, byte count/digest field, version and access rule. Mark facts not supplied as unknown. Evidence: four-row manifest. Rubric: 2 = raw/derived separation and honest provenance; 1 = object names but no scope/version; 0 = claims authenticated production export.
EX046 — Parser transcript. Objective: reproduce §12.2. Run the code, record the two exact output lines and explain why app line 3 and line 4 are rejected in that order. In a separate variant, change the duplicate A1 to A3 and predict accepted keys without modifying the original. Evidence: original/variant transcripts. Rubric: 2 = exact keys/reasons and variant separation; 1 = counts only; 0 = drops rejected rows.
EX047 — Join and outcome boundary. Objective: evaluate three claims: “I1 and A1 are definitely one session,” “I2 blocked A1,” and “r-601 committed.” For each, state supplied literal evidence, missing producer semantics and a bounded owner-approved request. Evidence: claim table. Rubric: 2 = literal s-7 versus validated join, s-8 distinction and queued-versus-commit limit; 1 = uncertainty without owner; 0 = declares a confirmed DB commit.
EX048 — Coverage design. Objective: make an absence test falsifiable. Propose an in-range positive control, an out-of-range negative control and a parser-rejection counter for the app source. Explain what a missing r-602 row can and cannot mean when A2 is rejected for bad time. Evidence: coverage card and failure interpretation. Rubric: 2 = source receipt, interval/path and rejection effect; 1 = only positive control; 0 = treats missing accepted row as proof of no request.
Subject synthesis
K12 produces four accepted and two rejected source rows while retaining three raw objects and one derived view. It does not authenticate the producer, order the events causally or prove a reservation commit. Chapter 13 uses that precise boundary to read two apparently contradictory alerts.
Continue studying — chapter 12 of 18
← Chapter 11 · Course chapter index · Chapter 13 →
Continue to the next authored lesson.