Test coverage by write route and by evidence layer | Professional Incident response and cyber resilience

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Day 2 — Test coverage by write route and by evidence layer

Learning decision. Decide whether a planned evidence source can support a particular decision, not whether “logging is on.” An audit producer may record scheduled jobs but omit manual edits; a collector may receive both but lose a join field; an export may filter out the correct source. NIST SP 800-92 describes log management as an enterprise process rather than a guarantee that every individual route is captured. This book's route matrix is a local rehearsal method, not a control requirement quoted from that publication.

Design separate, permitted training inputs: a scheduled S41 change with request train-17, a manual S41 change with request train-18, and an unrelated S40 change with request train-19. For each input, define what the producer should emit, what the collector should retain, and what the bounded export should return. A successful S41 selector must include the intended route's event and exclude S40; it must also preserve enough source metadata to explain why that row was selected. A missing event, absent request key and overbroad selection are three different defects, each with a different custodian question.

Lesson table; scroll horizontally to read all columns →

W1 rehearsal row, all hypothetical	Producer record	Collector record	Selected export	Defensible interpretation
Scheduled S41 / `train-17`	Slot and request key expected	Receipt time to measure	Matching S41 row expected	A pass would cover only this route and test window.
Manual S41 / `train-18`	Request key not yet established	Visibility unknown	Query result unknown	Cannot borrow the scheduled result.
Unrelated S40 / `train-19`	Distinct slot expected	Visibility to check	Must not match S41 selector	A false match falsifies selector specificity.

Worked example. Suppose a fictional rehearsal result shows train-17 at the producer and collector, but the export drops the request_id column. “Scheduled events are emitted” may be supported by the producer receipt; “the exported record can be joined to the reservation request” is not. If S40 also appears in the S41 export, narrowing the predicate is a query repair to test, not evidence that source logging was repaired. Conversely, an empty export cannot prove absence of a write until producer coverage, collector path, query semantics, clock conversion and permissions have been checked. Never fabricate these receipts for C01, whose training test remains only proposed.

Practice. Work EX002: write the six contract fields, state the authorized training tenant, exact start of a 20-minute clock, positive S41 event and S40 negative control, and name the evidence that would fail each layer. Compare with its worked answer. A test plan is complete when another authorized person could run it and recognize failure; a “green logging dashboard” is insufficient. Limit: even a passed trial samples one route, time and configuration. Previous: Day 1 · Week index · Next: Day 3.

Exercise for this lesson

EX002 — Evidence-readiness contract. Input: the proposed S41 scheduled-job change and unrelated S40 control. Produce all six contract fields, including expected fields, a 20-minute collection window and a falsifiable failure signal. State that the rehearsal requires a permitted training tenant. Use the EX001 rubric, with 2 requiring a negative control.

EX002 — Model contract. Decision question: can a responder retrieve a scheduled-job change to S41 within 20 minutes and distinguish it from unrelated S40? Expected producer: API audit, with slot ID, request ID, producer, UTC event time and collection time to verify. Time/retention: ask the custodian how long each event remains and start a 20-minute test clock at the authorized event. Access: application custodian in a permitted training tenant, with no personal enrollment data. Positive test: execute a synthetic authorized S41 change and recover its record. Negative control: query the same bounded selector and confirm unrelated S40 is absent; also check whether an alternate write route emits an event. Failure: missing request_id, wrong slot, missing event, inaccessible export or collection after 20 minutes. Fallback: retain the failed result, ask the owner for a narrower alternative source, and revise correlation plans. This is a test plan, not a successful test report.

Additional worked reasoning (EX002 — Falsifiable supervised rehearsal.): Define a permission-gated nonproduction change to S41, a known unrelated S40 event, the intended query predicate (exact slot plus stable request ID), expected producer UTC time, start at actual test event, twenty-minute maximum until a verified matching source receipt, and a separate observed query return. An approved test procedure is not its execution; the reviewer records both. Reject if S41 is absent, S40 is wrongly included, the ID is missing, the clock cannot be interpreted or the collection is late. The original C1 fixture's “yes” does not satisfy these conditions, and C1X's PASS_DECLARED_FIXTURE is only a consistency check. This prevents using a toy result to justify a real CTN collection.

Week 1 Day 2 extension — worked result. For W1 train-17, the producer must attest that a scheduled S41 event with that key was emitted; the collector must record receiving that exact source event; the restricted export must preserve key, slot, event time and query metadata. For train-18, repeat those checks on the manual route rather than inheriting the scheduled result. For train-19, verify that the S41 export excludes unrelated S40. If the export drops the key, the producer path might pass while decision-useful export coverage fails. If S40 appears, selection specificity fails even if the source is intact. If the producer never emitted a manual event, repairing a query will not repair producer coverage. Two points require all three layers, the negative control, and an explicit refusal to call a planned check an observed receipt.