Chapter 15 — Evidence and measurement
Lesson map
Lesson 15 published. Map, example, EX043.
flowchart TD
N0["Learning outcomes"]
N1["15.1 What counts as evidence"]
N2["15.2 Preservation"]
N3["15.3 Observation and inference"]
N4["15.4 Measurement"]
N5["15.5 Units and clocks"]
N6["15.6 Error and uncertainty"]
N7["15.7 Case notes"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX043"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["15.1 What counts as evidence"]
N2["15.2 Preservation"]
N3["15.3 Observation and inference"]
N4["15.4 Measurement"]
N5["15.5 Units and clocks"]
N6["15.6 Error and uncertainty"]
N7["15.7 Case notes"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX043"]
[Learning outcomes]
|
v
[15.1 What counts as evidence]
|
v
[15.2 Preservation]
|
v
[15.3 Observation and inference]
|
v
[15.4 Measurement]
|
v
[15.5 Units and clocks]
|
v
[15.6 Error and uncertainty]
|
v
[15.7 Case notes]
|
v
[Worked example] --> [Exercise EX043]
Learning outcomes
You will (1) preserve digital evidence with integrity; (2) measure SOC activities consistently; (3) distinguish observation, inference and conclusion; (4) record evidence in case notes.
15.1 What counts as evidence
In a SOC, evidence includes exported logs, alert records, screenshots of configuration and statements. Evidence must be preserved so its integrity can be shown later.
In the phishing case, Lucas's evidence included the sign-in export, the rule export, a screenshot of the rule's settings, the original phishing message and a short statement from s2047's owner confirming she had entered her password on the fake page. Each item received a reference such as E-3 so case notes could cite it precisely.
15.2 Preservation
Exports are saved read-only with a hash (a digital fingerprint). If a file changes, its hash changes. Case notes record who exported what, when and from where.
Lucas computes a SHA-256 hash of each export as soon as it is saved and writes it in the case notes. If anyone later questions whether the sign-in export was edited, recomputing the hash and comparing it with the recorded value answers the question in seconds. Working copies are made for analysis so the original is never opened for editing.
15.3 Observation and inference
"Sign-in from 198.51.100.x at 02:14" is an observation. "The account was compromised" is an inference. Case notes keep them separate and show the reasoning.
Keeping them apart lets a reader challenge the reasoning without disputing the facts. If someone suggests s2047's owner was simply travelling, the observations stay the same, a sign-in from a Dutch hosting address and an external forwarding rule six minutes later, and the discussion focuses on whether the inference still holds. Here, the owner's statement settled it.
15.4 Measurement
Consistent measures — time to acknowledge, time to contain, number of accounts affected — let the SOC compare incidents and improve. Definitions must be fixed: does "contain" mean password reset or all sessions revoked?
Riverton defines containment as both a password reset and revocation of all sessions, since a reset alone leaves an attacker's existing session working. Under that definition, s2047 was contained at 08:41 and s3310 at 11:20. Recording the definition beside each figure prevents a later report from comparing times measured in different ways.
15.5 Units and clocks
All times in UTC; durations in minutes. Mixed units cause errors.
The campaign shows why: delivery at 01:50, the first compromise at 02:14 and containment of s3310 at 11:20 come from different systems, and a report mixing local and UTC times would misstate how long the attacker had access. Writing durations in minutes, such as the gap between sign-in and rule, avoids confusion between hours and minutes. The worked results show the bug this prevents: a comparison of minutes that ignored hours broke matches across an hour boundary. Lucas's tools now parse every timestamp into a full datetime in UTC before doing any arithmetic, and reports state the zone beside each time.
15.6 Error and uncertainty
Some measures are uncertain: the number of people who clicked is estimated from DNS lookups and reports. Lucas reports a range and its basis.
For the number of people who clicked, the 41 campus DNS lookups give a lower bound, since off-campus clicks are not logged. Reports and sign-in evidence give other partial views. Lucas's case note states a range with its basis rather than a single figure, and explains that the true number is likely to be higher than any one source suggests.
15.7 Case notes
Each step in the campaign response is recorded as it happens, not reconstructed afterwards.
Notes written during the response captured details that would have been lost later, such as the exact time the forward rule was removed and who removed it. When the review chapter builds a timeline of the campaign, it relies on these entries. Reconstructing from memory would have produced plausible but uncertain times that could not be checked against any source. Lucas keeps his notes in the case system, which timestamps each entry automatically.
Chain of custody
When evidence may be needed by the university's disciplinary process or by police, a chain-of-custody record lists every transfer of the evidence with time and person. Without it, the evidence may be challenged even if it is accurate.
Timeliness
Notes written hours later lose detail; Lucas writes them while working.
Worked example
Lucas's case note: "02:14 UTC sign-in s2047 from NL hosting address (observation, export E-3, hash recorded). 02:20 rule created forwarding to external address (observation, E-5). Inference: account compromised (supported by E-3, E-5, owner statement). Contained 08:41 (password reset and sessions revoked)."
Synthetic case. Notes and exports are invented.
Errors and limits
Preserved evidence shows what logs recorded, not everything that happened.
Exercises
- EX043 — Observation versus inference. Classify "the account was compromised". Rubric: 3 = inference, supported by observations; 2 = inference only; 1 = observation; 0 = none.
- EX044 — Containment definition. Why must "contained" be defined precisely? Rubric: 3 = different actions mean different times; consistent comparison; 2 = partial; 1 = no reason; 0 = none.
- EX045 — Click estimate. How should the click count be reported? Rubric: 3 = range with basis (DNS lookups, reports); 2 = partial; 1 = exact number; 0 = none.
Synthesis
Preserve evidence with integrity, measure consistently and keep observations separate from inferences.
Continue studying — chapter 15 of 40
← Chapter 14 (draft) · Course chapter index · Chapter 16 (draft) →