Chapter 9 — Evidence and measurement
Lesson map
Lesson 09 published. Map, example, EX025.
flowchart TD
N0["Learning outcomes"]
N1["9.1 Baselines"]
N2["9.2 Z-scores"]
N3["Lab — baseline and deviations"]
N4["9.3 Reading the result"]
N5["9.5 Evidence value"]
N6["9.6 Tuning"]
N7["9.7 Integrity"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX025"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["9.1 Baselines"]
N2["9.2 Z-scores"]
N3["Lab — baseline and deviations"]
N4["9.3 Reading the result"]
N5["9.5 Evidence value"]
N6["9.6 Tuning"]
N7["9.7 Integrity"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX025"]
[Learning outcomes]
|
v
[9.1 Baselines]
|
v
[9.2 Z-scores]
|
v
[Lab — baseline and deviations]
|
v
[9.3 Reading the result]
|
v
[9.5 Evidence value]
|
v
[9.6 Tuning]
|
v
[9.7 Integrity]
|
v
[Worked example] --> [Exercise EX025]
Learning outcomes
You will (1) build simple baselines of normal activity; (2) detect deviations with a z-score; (3) understand the limits of statistical alerts; (4) record measurements as evidence.
9.1 Baselines
Normal activity varies. Counting failed logons per hour over two weeks gives a baseline mean and spread for a server.
Two weeks is enough to show daily rhythms but may miss monthly events such as password expiry days. Jonah records the period, the server and the log source with the baseline, and plans to refresh it monthly. A baseline built during an undetected attack would treat that attack as normal, so he also checks the period for known incidents. The baseline is stored with the case so later comparisons use the same reference.
9.2 Z-scores
A z-score measures how many standard deviations an observation is from the mean. A high z-score marks unusual activity worth checking.
In the lab the baseline averages four failures an hour with a spread of just over one, so an hour with 9 failures sits nearly four standard deviations above normal. The threshold of 3 is a choice: lower thresholds catch more but alert more often on harmless variation. Jonah writes the threshold and the reason for it next to the alert rule. Changing the threshold later requires a recorded reason, so alerts from different months can still be compared fairly.
Lab — baseline and deviations
"""Baseline of failed logons per hour and a simple z-score check."""
import statistics as st
history = [3, 5, 4, 2, 6, 4, 3, 5, 4, 4, 3, 5, 6, 2]
mu, sd = st.mean(history), st.stdev(history)
for obs in (5, 9, 41):
z = (obs - mu) / sd
print(f"observed={obs:3} z={z:5.1f} {'ALERT' if z > 3 else ''}")
print(f"baseline mean={mu:.2f} sd={sd:.2f}")
Run with python3 lab_baseline.py. Verified output:
observed= 5 z= 0.8
observed= 9 z= 3.8 ALERT
observed= 41 z= 28.4 ALERT
baseline mean=4.00 sd=1.30
9.3 Reading the result
The baseline is a mean of 4.00 failed logons per hour with standard deviation 1.30. An hour with 5 failures gives z = 0.8 (normal). An hour with 9 gives z = 3.8 and an hour with 41 gives z = 28.4; both exceed the threshold of 3 and alert. The 41 is the hour of the guessing against the server from 10.20.4.12.
Jonah also records which source supplied the 41 failures: 38 came from the same desktop address seen logging on to CLERK-03. That link is what makes the alert useful for the case rather than just a statistic. Jonah's report keeps the statistic and the source together, so the alert can be traced to a specific desktop. Without the source, a high score would only say that something unusual happened on the file server, not where to look next.
9.4 Limits
Real counts are not perfectly normal; holidays, password-expiry days and new software skew them. A z-score threshold is a rule of thumb, not a probability statement.
The 9-failure hour shows the limit in practice: it alerted, and the cause was a staff member with an expired password. A high score says only that an hour is unusual compared with the baseline. Jonah annotates each alert with the cause once found, which over time shows how often the rule points at harmless events.
9.5 Evidence value
The baseline and the alerting hour are recorded with their data source, period and calculation, so the claim "unusually high" can be checked.
Jonah's sentence about FILESRV-02 shows the form: the period, the mean and spread, the alerting hour with its count and score, and the source address of most failures. Anyone with the preserved logs and the script can reproduce each number. A claim that activity was unusual without these details cannot be tested or defended. The source address comes from the same log, not from memory.
9.6 Tuning
Thresholds that alert too often get ignored; review alerts monthly and adjust with records of why.
After the CLERK-03 case, the monthly review of the failed-logon rule would note the expired-password alert as a known benign cause and consider whether password-expiry days need a separate baseline. The change and its reason are recorded, so the rule's history explains why it behaves as it does. A rule that nobody tunes either floods analysts or is quietly disabled, and both outcomes lose evidence. Each change to a threshold should state the alerts it would have suppressed, so the cost of tuning is visible.
9.7 Integrity
Measurements derived from logs are recomputed from preserved log copies.
If a baseline or z-score is questioned, Jonah recomputes it from the preserved, hashed log export rather than from a dashboard, which may have changed or aged out. The recomputation should match exactly; if it does not, the difference must be explained before the measurement is used in any report. Measurements quoted in reports must cite the export's hash. For the FILESRV-02 figures, that means the case folder holds the hashed export for the fourteen baseline days and the alerting hour, the script that computed the score and its version, so a reviewer can reproduce the 41 failures and the score without access to the live server.
Choosing what to baseline
Not everything needs a baseline. Jonah picks measures linked to likely attacks: failed logons per server, new services per week, outbound data volume per host. Each has a named owner who reviews alerts, so measures do not pile up unread.
Worked example
Jonah writes: "Failed logons on FILESRV-02 averaged 4.0 per hour (sd 1.3) over 14 days; on 4 May 08:00–09:00 UTC there were 41, z = 28.4. The source of 38 of these was 10.20.4.12."
Synthetic case. Data come from the lab script.
Errors and limits
A high z-score shows unusual activity, not its cause; the 9-failure hour turned out to be a staff member with an expired password.
Exercises
- EX025 — Lab z-scores. Report the three z-scores and which alert. Rubric: 3 = 0.8, 3.8, 28.4; 9 and 41 alert; 2 = partial; 1 = wrong; 0 = none.
- EX026 — Cause limit. What did the 9-failure hour turn out to be? Rubric: 3 = expired password; shows z ≠ cause; 2 = partial; 1 = attack; 0 = none.
- EX027 — Threshold nature. Why is z > 3 a rule of thumb? Rubric: 3 = data not normal; skewed by events; 2 = partial; 1 = exact probability; 0 = none.
Synthesis
Baselines turn "unusual" into a checkable measurement; investigation still finds the cause.
Continue studying — chapter 09 of 30
← Chapter 08 (draft) · Course chapter index · Chapter 10 (draft) →