Chapter 7 — Event-source mapping and log retention

Chapter 7 — Event-source mapping and log retention

Lesson map

Lesson 07 published. Map, example, EX025.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["7.1 Ask what the source can decide"]
  N2["7.2 Calculate, then challenge, an expiry hypothesis"]
  N3["7.3 Build a bounded preservation queue"]
  N4["7.4 Coverage test as a small experiment"]
  N5["Subject synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX025"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["7.1 Ask what the source can decide"]
  N2["7.2 Calculate, then challenge, an expiry hypothesis"]
  N3["7.3 Build a bounded preservation queue"]
  N4["7.4 Coverage test as a small experiment"]
  N5["Subject synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX025"]

[Learning outcomes and connection]
    |
    v
[7.1 Ask what the source can decide]
    |
    v
[7.2 Calculate, then challenge, an expiry hypothesis]
    |
    v
[7.3 Build a bounded preservation queue]
    |
    v
[7.4 Coverage test as a small experiment]
    |
    v
[Subject synthesis]
    |
    v
[Worked example] --> [Exercise EX025]

Learning outcomes and connection

The learner will map a decision question to producers and fields, distinguish event, ingestion, collection and query time, and prioritize a short-lived source without turning a retention label into a guaranteed deletion deadline. Chapter 6 posed route and control questions; Chapter 7 constructs a request plan that could answer them. Chapter 8 will test how missing or delayed records affect competing failure hypotheses. NIST SP 800-92 is older, high-level guidance for log-management processes; this chapter's source inventory and clock exercise are original, not prescribed by NIST.

7.1 Ask what the source can decide

For a response question such as “Did the API receive r-201?” the candidate sources are not interchangeable. The edge can report that it returned 503 and labelled an upstream timeout. An API access log may show receipt under a different upstream ID. An IdP record can show a session decision but may not include the application request ID. A host process record may show a worker start but not the request it issued. The first step is a source matrix: decision question, producer, expected identifying fields, interval, owner, clock basis, collection path, retention anchor and a known gap. A later query result should cite the matrix row that makes it relevant.

The source may fail to answer the question for five different reasons: the event was never emitted; it was emitted but not transported; it arrived late; it was stored but omitted by the query; or it expired or became inaccessible. “No row found” is ambiguous until coverage and ID semantics are tested. A query should use both an in-range positive control and a known out-of-range or absent negative control when permitted. Record the query timestamp and filter. An empty API sample from Chapter 2 cannot settle whether r-201 reached the API, particularly because the gateway may rewrite IDs. Chapter 4's rejected-row report is another coverage limit: a source record may be present but unusable under the selected schema.

The inventory below is L7, a separate fictional retention-planning branch. It does not assert that the main case has these retention policies or that any log was exported. Each duration is a classroom assumption with an explicit anchor; the custodian must verify both in an organization. A “10-minute” statement without an anchor is not enough to calculate an expiry. An identity source could retain a detail row from event time, ingestion time or last access, producing different deadlines.

At an exercise planning time of 12:06 UTC on 11 October 2026, identity detail has the shortest calculated timed window, but the host volatile source has an unknown deadline and may need simultaneous owner contact. The table does not authorize collection. The priority is to ask the relevant custodians and incident lead for a narrow, approved preservation path, with a privacy limit and exact fields. If that cannot be arranged before a proposed expiry, record the loss risk and escalate; do not run an unauthorized bulk export.

7.2 Calculate, then challenge, an expiry hypothesis

Run this Python 3 code offline. It calculates three expiry hypotheses from literal UTC timestamps. The code does not check whether a logging system actually retains data for that long, whether clocks are synchronized or whether a legal hold overrides normal disposal. Its result is a planning aid that must be confirmed with the custodian.

from datetime import datetime, timedelta

specs = [
    ("identity_detail", "2026-10-11T11:58:00+00:00", 10),
    ("edge_ring", "2026-10-11T12:00:05+00:00", 15),
    ("api_audit", "2026-10-11T12:00:04+00:00", 1440),
]
for name, anchor, minutes in specs:
    expiry = datetime.fromisoformat(anchor) + timedelta(minutes=minutes)
    print(name, expiry.strftime("%Y-%m-%dT%H:%M:%SZ"))

Expected output: identity_detail 2026-10-11T12:08:00Z, edge_ring 2026-10-11T12:15:05Z, api_audit 2026-10-12T12:00:04Z. From 12:06:00 UTC the first calculated window is two minutes. That is not a “two-minute permission” or a certified retention promise. If the identity row was ingested at 12:01 rather than emitted at 11:58, a source that anchors on ingestion would yield a different expiry. Record that as a variant, not as a correction to L7. A source clock offset could similarly move an apparent event-time anchor. Never silently replace the raw time.

7.3 Build a bounded preservation queue

A queue item must tell an authorized collector exactly what to request and why. Example Q7-1 asks the identity custodian, under an incident-lead-approved case scope, to preserve only the detailed session rows for the L7 principal/session identifier during 11:57:30–11:59:30 UTC, including producer event time, ingestion time, source object ID, session semantics and collection receipt. It asks the custodian to confirm the retention anchor and source clock. It expressly excludes unrelated accounts and broad user attributes. The request is pending until the custodian and authorized role accept it.

Q7-2 asks the gateway custodian for a bounded 12:00:00–12:00:10 UTC mapping of r-201 to any upstream ID and the upstream timeout fields, plus source coverage and producer time. Q7-3 asks the application custodian for an API audit sample covering the same interval, filtered only after documenting possible ID rewriting. These requests might be pursued in parallel if owners and authority exist. Their order should reflect disappearance risk, decision value and collection cost, not just which tool is easiest. A short-retention source with no decision relevance should not automatically outrank a less volatile source that can settle an urgent containment question.

An answer can be unavailable as well as positive or negative. If the identity custodian reports that the detail row expired, record the reported time, source, query and coverage; do not fill the gap with a guess. If an export arrives, carry forward its raw-byte receipt and transformation report from Chapters 4–5. If only a screenshot arrives, record its limited provenance and seek a source record where appropriate. A screenshot can show an interface at capture time but may omit filters and earlier data. The analyst's timeline should retain both source event time and the later time when the analyst learned of it.

7.4 Coverage test as a small experiment

Suppose a later separate L7 test has a known synthetic control event at 12:00:06 that the API custodian says was emitted. Querying the exact 12:00:00–12:00:10 window should return it if the pipeline, fields and access path are working. If it does not, the missing r-201 row is even less informative. If the control appears, that improves confidence in this interval and source but does not prove complete coverage of all routes or that the request ID was preserved unchanged. The test should document the control's producer receipt, not merely an analyst-created string in a spreadsheet. A negative control outside the interval helps reveal accidental overbroad queries. Neither control should be run against production without permission.

Common errors and limits

• Treating a retention duration without an anchor as an exact deletion deadline.
• Equating event time, ingestion time, collection time and query-run time.
• Searching by one request ID without checking whether the gateway rewrites it.
• Letting a synthetic control event become a main-case observation.
• Prioritizing easy collection over decision value and disappearance risk.

Exercises — subject 7

EX025 — Source matrix. Objective: map three decision questions to producers. For “did r-201 reach API?”, “did a session exist?” and “did a worker process start?”, give source, expected fields, custodian, clock basis, retention question and one absence interpretation that remains unresolved. Evidence: three-row matrix. Rubric: 2 = producer limits and ID mapping explicit; 1 = sources listed without absence conditions; 0 = turns a missing sample row into a universal fact.

EX026 — Expiry transcript and variant. Objective: reproduce §7.2. Give the three exact output lines and calculate time remaining for identity detail at 12:06:00 UTC. Then assume only in a variant that identity retention starts from ingestion at 12:01:00; compute its 10-minute expiry and explain why this does not rewrite L7. Evidence: two calculations. Rubric: 2 = correct dates/times and anchor caveat; 1 = one arithmetic error; 0 = calls the estimate permission to export.

EX027 — Priority queue. Objective: write actionable, lawful requests. At 12:06, rank identity detail, host volatile state, edge ring and API audit for initial owner contact. Give a reason and a bounded query for the first two, distinguish unknown host expiry from timed retention, and state what to record if one source is unavailable. Evidence: queue with owner and status fields. Rubric: 2 = risk and decision value together, approval pending; 1 = ranked list without scope; 0 = instructs a broad export.

EX028 — Coverage experiment. Objective: distinguish absence from no event. Design the L7 in-range and out-of-range controls for a 12:00:00–12:00:10 query. State what each outcome supports and what it cannot prove about r-201 or unobserved routes. Evidence: experiment card. Rubric: 2 = controls, producer receipt and bounded inference; 1 = positive control only; 0 = treats a successful control as proof of total completeness.

Subject synthesis

Source selection is a technical and authorization decision. The expiry script makes assumptions inspectable; it does not create or extend access. Chapter 8 will use source limits and uncertain clock ordering to compare a network fault, identity denial, host restart and maintenance without inventing a culprit.

Continue studying — chapter 07 of 18

← Chapter 06 · Course chapter index · Chapter 08 →

Continue to the next authored lesson.