Chapter 24 — Evidence pipeline and decision-support methods
Lesson map
Lesson 24 published. Map, example, EX047.
flowchart TD
N0["Learning outcomes and connection"]
N1["24.1 The evidence path"]
N2["24.2 Inspectable scoring"]
N3["24.3 Omission is a decision"]
N4["24.5 Human review and rollback"]
N5["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX047"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["24.1 The evidence path"]
N2["24.2 Inspectable scoring"]
N3["24.3 Omission is a decision"]
N4["24.5 Human review and rollback"]
N5["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX047"]
[Learning outcomes and connection]
|
v
[24.1 The evidence path]
|
v
[24.2 Inspectable scoring]
|
v
[24.3 Omission is a decision]
|
v
[24.5 Human review and rollback]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX047]
Learning outcomes and connection
After this chapter you can (1) design an evidence path from ingestion through validation, scoring and presentation; (2) make triage recommendations inspectable — every score explained by its inputs; (3) test how omissions in the pipeline change a responder's decision; and (4) design human review and rollback for scoring changes. It builds on Chapters 9, 14 and 22 and prepares Chapters 29, 30 and 36.
24.1 The evidence path
Stages: ingest (receive, hash raw, store); validate (schema, timestamps, quarantine with reasons); normalise and enrich (entity resolution, context such as asset criticality); score (rules or models producing a priority); present (queue with explanations); decide (human, recorded); feedback (decision outcomes returned to tune scoring). Each stage has an input contract, output contract, error path and metrics. Provenance links each presented item to its raw inputs. At HVSS, the v3.2 regression broke the contract between ingest and normalisation without any stage reporting an error.
24.2 Inspectable scoring
A score that cannot be explained cannot be challenged. For rule-based scoring, present the contributing factors with their weights ("+30: MFA reset within 1 h of new-location sign-in"). For statistical models, present the main features and the training data's period; Arp et al. (2022) catalogue pitfalls — data snooping, unrealistic base rates, spurious correlations — that make security models look better in evaluation than in operation. Whatever the method, the responder must see what was not available: missing enrichment, dropped fields, delayed sources.
24.3 Omission is a decision
When a pipeline drops data, it makes a decision on the responder's behalf without telling them. The most dangerous omissions are silent: unknown fields discarded, late events excluded from correlation windows, enrichment failures treated as "not critical".
Lab 24 makes such a hidden decision visible by printing the ranking both ways. With unknown fields dropped, the pipeline silently decided that mfa_reset and grantor_is_self did not matter for user u17, and A2 moved to the top of the queue. Nothing in the dropped-field ranking told the responder that two inputs were missing. A line such as "fields dropped: 2" on A1 and A3 would have turned that hidden decision into information the responder could weigh.
24.4 Worked example (synthetic): ranking with and without unknown fields
HVSS's ranking adds weight for two vendor fields introduced recently. Lab 24 reproduces the effect of the v3.2 ingest regression (Chapter 14).
"""Lab 24 (synthetic): alert ranking when the normaliser drops unknown fields."""
alerts = [
{"id": "A1", "rule": "impossible_travel", "user": "u17", "base": 40, "mfa_reset": True},
{"id": "A2", "rule": "malware_quarantined", "user": "u02", "base": 55},
{"id": "A3", "rule": "new_admin_grant", "user": "u17", "base": 50, "grantor_is_self": True},
{"id": "A4", "rule": "failed_logins", "user": "u31", "base": 30},
]
KNOWN = {"id", "rule", "user", "base"}
def normalise(a, keep_unknown):
out = {k: v for k, v in a.items() if k in KNOWN}
extra = {k: v for k, v in a.items() if k not in KNOWN}
if keep_unknown: out["extra"] = extra
return out, sorted(extra)
def score(a):
s = a["base"]; x = a.get("extra", {})
if x.get("mfa_reset"): s += 30
if x.get("grantor_is_self"): s += 35
return s
for keep in (False, True):
rows, dropped = [], []
for a in alerts:
n, ex = normalise(a, keep); rows.append((score(n), n["id"])); dropped += ([] if keep else ex)
rows.sort(reverse=True)
print(("keep" if keep else "drop") + " unknown fields: ranking", [f"{i}:{s}" for s, i in rows],
"| dropped:", dropped or "none")
Expected output:
drop unknown fields: ranking ['A2:55', 'A3:50', 'A1:40', 'A4:30'] | dropped: ['mfa_reset', 'grantor_is_self']
keep unknown fields: ranking ['A3:85', 'A1:70', 'A2:55', 'A4:30'] | dropped: none
With fields dropped, a quarantined-malware alert (already handled by the endpoint product) ranks first and the self-granted administrator role (A3) and the impossible-travel login followed by an MFA reset (A1) — both pointing to the same user u17 and together a strong account-takeover signal — rank second and third. With fields kept, they rank first and second. A responder working top-down with limited time could plausibly handle A2 first and lose the most important minutes. The pipeline fix is preservation (Chapter 14); the decision-support fix is to show "fields dropped: 2" on every affected alert and to group alerts by entity so that A1 and A3 appear together.
24.5 Human review and rollback
Scoring changes are deployments: run new and old scorers in parallel on live data for a period, compare rankings (rank correlation, top-k overlap) and examine every alert whose rank moved by more than a set amount. Keep the previous scorer deployable; record scorer version in every presented alert so that past decisions can be understood in the context of the ranking the responder actually saw. Applied to Lab 24, a parallel run would flag A3 moving from 85 to 50 and A1 from 70 to 40 under v3.2.
24.6 Errors and limits
- Scores without explanations.
- Silent omission of fields, late events or enrichment.
- Alerts ranked individually when entities link them.
- Scorer changes without parallel comparison.
Every error in this list appears in the Lab 24 ranking. The dropped-field scores of 50 and 40 for A3 and A1 carried no explanation of what was missing. The two vendor fields vanished silently, with no counter or warning on the affected alerts. The two u17 alerts were ranked separately, so a strong account-takeover pattern looked like two medium alerts. The v3.2 normaliser also went live without its rankings being compared against the previous version on live data.
Exercises — chapter 24
EX047 — Pipeline checks. Objective: specify checks per stage. Inputs: the seven stages of 24.1. Steps: for each stage, one check, its failure route and its metric. Expected evidence: table. Rubric: 3 = seven specific checks with routes and metrics including field-coverage; 2 = five; 1 = generic; 0 = none.
EX048 — Entity grouping. Objective: change decision support. Inputs: Lab 24 with fields kept. Steps: modify the code to aggregate scores per user (sum or max with a co-occurrence bonus of +20 when two rules fire for one user within the batch) and print an entity ranking. Expected evidence: code and output. Rubric: 3 = correct ranking with u17 first and justified aggregation choice; 2 = output only; 1 = attempt; 0 = none.
Chapter synthesis
An evidence pipeline is a chain of contracts from raw record to decision. Make scores explainable, make omissions visible, group by entity and treat scorer changes as deployments with parallel comparison and rollback.
Continue studying — chapter 24 of 50
← Chapter 23 · Course chapter index · Chapter 25 →
Continue to the next authored lesson.