Chapter 39 — Capstone evaluation
Lesson map
Lesson 39 published. Map, example, EX115.
flowchart TD
N0["Learning outcomes"]
N1["39.1 Evaluation design"]
N2["Lab — detection rates with intervals"]
N3["39.2 Reading the result"]
N4["39.3 Misses"]
N5["39.4 Detection times"]
N6["39.5 False alerts"]
N7["39.6 Requirements summary"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX115"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["39.1 Evaluation design"]
N2["Lab — detection rates with intervals"]
N3["39.2 Reading the result"]
N4["39.3 Misses"]
N5["39.4 Detection times"]
N6["39.5 False alerts"]
N7["39.6 Requirements summary"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX115"]
[Learning outcomes]
|
v
[39.1 Evaluation design]
|
v
[Lab — detection rates with intervals]
|
v
[39.2 Reading the result]
|
v
[39.3 Misses]
|
v
[39.4 Detection times]
|
v
[39.5 False alerts]
|
v
[39.6 Requirements summary]
|
v
[Worked example] --> [Exercise EX115]
Learning outcomes
You will (1) evaluate the capstone against its requirements; (2) compute detection rates with confidence intervals; (3) analyse false alerts and detection times; (4) state limitations.
39.1 Evaluation design
The 20 authorised scenarios are run before (existing rules) and after (new package). Each scenario is scored detected or missed, with detection time.
Running the same twenty scenarios twice, before and after, means differences come from the rules, not the scenarios. The scenarios include the two slow attackers added after the seminar. Detection time is measured from the triggering action to the alert, as the requirement states, using the harness timestamps added earlier. Lucas ran both sets on the same day with the same test accounts.
Lab — detection rates with intervals
"""Wilson 95% interval for the capstone detection rate on authorised test scenarios (synthetic)."""
import math
def wilson(k, n, z=1.96):
p = k / n; d = 1 + z*z/n
c = (p + z*z/(2*n)) / d; h = z*math.sqrt(p*(1-p)/n + z*z/(4*n*n)) / d
return c - h, c + h
for label, k, n in (("before", 11, 20), ("after", 18, 20)):
lo, hi = wilson(k, n)
print(f"{label:6} detected {k}/{n} = {k/n:.2f} 95% CI [{lo:.2f}, {hi:.2f}]")
Run with python3 lab_wilson.py. Verified output:
before detected 11/20 = 0.55 95% CI [0.34, 0.74]
after detected 18/20 = 0.90 95% CI [0.70, 0.97]
39.2 Reading the result
Before: 11 of 20 detected (0.55, 95% interval 0.34–0.74). After: 18 of 20 (0.90, interval 0.70–0.97). The intervals overlap only slightly; with 20 scenarios, uncertainty is large, but the improvement is consistent with the design.
The slight overlap means the evidence for improvement is good but not overwhelming. With twenty scenarios, each one moves the rate by five points. Lucas states the improvement as seven more scenarios detected, which is concrete, and leaves the intervals to show how precise the rates are. He also reports which individual scenarios changed from missed to detected.
39.3 Misses
The two missed scenarios used a forwarding rule to an internal account that then forwarded externally — a pattern none of the rules covered. They become future work.
The pattern is a two-step chain: the attacker forwards to an internal account it also controls, which then forwards outside. No single rule sees an external forward from the compromised account. Lucas describes how a future rule could follow such chains through the mailbox audit log. He does not adjust the package to catch them before the evaluation ends, since that would mean tuning to the test.
39.4 Detection times
Median detection time was 14 minutes; the slowest was 27 minutes, within the 30-minute requirement.
Times came from the harness, so they are exact for test scenarios. The slowest included the deliberate wait for delayed mail audit events. Live times may differ, since real activity arrives with variable delays, which the single live detection in twenty-five minutes illustrates. Lucas reports the full list of times in an appendix rather than only the median and the maximum, so readers can see whether most detections clustered near the median or were spread across the window. Times for the slow-attacker scenarios are marked.
39.5 False alerts
Over four weeks live, false alerts averaged 1.5 per week, below the three-per-week requirement.
Each false alert was reviewed: most were staff with approved forwarding exceptions who signed in from abroad. Lucas added those exceptions to the rule's known benign matches. The rate is an average over four weeks, so a busy week could still exceed three, which the SOC will watch. Lucas reports the average as one and a half per week alongside the weekly counts, so the spread is visible. Analysts confirmed in the feedback described below that this level did not distract them from other work.
39.6 Requirements summary
Detection ≥80%: met (90%, though the interval's lower bound is 70%). False alerts ≤3/week: met. Within 30 minutes: met.
The detection line states both the result and the caveat, as the worked example below explains. Presenting the caveat beside the claim, rather than in a footnote, lets readers see in one place what was met and how confidently. Lucas also lists the requirement on detection time with the median and the slowest value. A summary table with one row per requirement, the result and a brief caveat lets a reader see the outcome in seconds.
Analyst feedback
Lucas asks the six student analysts whether the new alerts were clear. Five say the alert text, which lists the risky sign-in, the rule created and the recipients, let them decide quickly; one asks for a link to the user's sign-in history, which he adds. Analysts' experience is part of evaluation: a rule that detects but confuses slows response.
Comparison with the baseline period
In the four weeks before the package, two real compromises were found by user reports after several hours. In the four weeks after, one real compromise was detected by the package in 25 minutes.
Worked example
Lucas writes: "The package met all three requirements on test scenarios and four weeks of operation. Because the lower confidence bound for detection is 0.70, below the 0.80 target, we cannot claim the target is met with confidence; more scenarios are needed."
Synthetic case. Results come from the lab script and invented operation data.
Errors and limits
Test scenarios approximate attacks; operational detection rates may differ.
Exercises
- EX115 — Lab intervals. Report before and after detection with intervals. Rubric: 3 = 0.55 [0.34, 0.74]; 0.90 [0.70, 0.97]; 2 = one; 1 = wrong; 0 = none.
- EX116 — Missed pattern. What pattern did the missed scenarios use? Rubric: 3 = internal forward then external; 2 = partial; 1 = unknown; 0 = none.
- EX117 — Confidence caveat. Why can Lucas not claim the 80% target is met with confidence? Rubric: 3 = lower bound 0.70 below 0.80; more scenarios needed; 2 = partial; 1 = he can; 0 = none.
Synthesis
Evaluate against each requirement with intervals and honest caveats, and turn misses into future work.
Continue studying — chapter 39 of 40
← Chapter 38 (draft) · Course chapter index · Chapter 40 (draft) →