Chapter 27 — Security operations and blue team: verification
Lesson map
Lesson 27 published. Map, example, EX079.
flowchart TD
N0["Learning outcomes"]
N1["27.1 What verification checks"]
N2["27.2 Coverage honestly measured"]
N3["Lab — coverage status"]
N4["27.3 Reading the result"]
N5["27.4 Planning"]
N6["27.5 Test design"]
N7["27.6 Recording"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX079"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["27.1 What verification checks"]
N2["27.2 Coverage honestly measured"]
N3["Lab — coverage status"]
N4["27.3 Reading the result"]
N5["27.4 Planning"]
N6["27.5 Test design"]
N7["27.6 Recording"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX079"]
[Learning outcomes]
|
v
[27.1 What verification checks]
|
v
[27.2 Coverage honestly measured]
|
v
[Lab — coverage status]
|
v
[27.3 Reading the result]
|
v
[27.4 Planning]
|
v
[27.5 Test design]
|
v
[27.6 Recording]
|
v
[Worked example] --> [Exercise EX079]
Learning outcomes
You will (1) verify that detections work as specified; (2) measure ATT&CK coverage honestly; (3) distinguish tested from untested rules; (4) plan verification for gaps.
27.1 What verification checks
Verification checks that each rule fires on the behaviour it targets and stays quiet otherwise. It uses safe synthetic events in a test environment or approved test accounts in production.
For the correlation rule, verification means a test account signs in from a test address flagged as a new country, creates a forwarding rule, and the SOC confirms an alert appears with both events. A second test creates an ordinary folder rule after the same sign-in and confirms no alert appears.
27.2 Coverage honestly measured
Coverage is the share of priority ATT&CK techniques with a tested detection. A rule that exists but has never been tested is not counted as coverage, because it may silently fail.
Counting untested rules would have hidden the broken email collection rule found in the worked example below: it existed, appeared on the coverage chart and had not matched anything for months, because the field it relied on had been renamed. Honest counting turned a reassuring number into a list of work.
Lab — coverage status
"""ATT&CK technique coverage for the campus SOC (synthetic status per technique)."""
status = {"T1566 Phishing": "tested", "T1078 Valid Accounts": "tested", "T1110 Brute Force": "tested",
"T1114 Email Collection": "rule-untested", "T1098 Account Manipulation": "rule-untested",
"T1021 Remote Services": "none", "T1486 Data Encrypted for Impact": "tested", "T1048 Exfiltration Over Alternative Protocol": "none"}
n = len(status)
for k in ("tested", "rule-untested", "none"):
items = [t for t, s in status.items() if s == k]
print(f"{k:13} {len(items)}/{n} = {len(items)/n:.0%}")
print("gaps:", [t.split()[0] for t, s in status.items() if s == "none"])
Run with python3 lab_coverage.py. Verified output:
tested 4/8 = 50%
rule-untested 2/8 = 25%
none 2/8 = 25%
gaps: ['T1021', 'T1048']
27.3 Reading the result
Of eight priority techniques, four (50%) have tested detections, two (25%) have rules not yet tested, and two (25%) — remote services (T1021) and exfiltration over alternative protocols (T1048) — have nothing.
The figures describe Riverton's SOC fairly: the techniques seen in the phishing campaign and the ransomware exercise have tested detections, while those involving research servers and DNS have none. The gaps reflect where the SOC's data sources are weakest, as the data chapter showed, rather than where attackers are least likely to go.
27.4 Planning
Untested rules are tested first because it is cheap. The two gaps need data sources: remote-service logons on research servers and DNS-based exfiltration detection using resolver logs.
Testing the two untested rules needed only a test account and an afternoon. Filling the gaps needs more: research servers must forward remote-service logons reliably, and DNS exfiltration detection needs resolver logs with enough detail to spot unusually long or frequent queries. Lucas's plan lists the owner and the first step for each.
27.5 Test design
Each test states input, expected result and pass criteria. For mailbox rules, a test account creates a forwarding rule after a simulated risky sign-in from a test address.
For the mailbox test, the pass criterion is an alert within the window plus the audit delay, naming the test account and listing the rule. A test that produced an alert without the rule details would fail, since analysts need both events to triage quickly. Stating criteria in advance prevents a partial result from being counted as a pass.
27.6 Recording
Results are stored with date, rule version and tester, and repeated each term.
Repeating each term catches silent breakage like the renamed field. Storing the rule version with each result shows which version passed, so if a later change breaks the rule, the last working version can be found. The record also shows the next tester exactly what was done before. For the correlation rule, the first record lists the test account, the sign-in address, the rule created, the alert time and the tester's name. The second, a term later, repeated the same steps after the window change and confirmed the new version still fired.
Who tests
Rules are tested by someone other than their author where possible. An author may unconsciously build a test that matches their own assumptions; a second person tests what the rule claims to detect.
Worked example
Testing the two untested rules finds one working (account manipulation alert) and one broken: the email collection rule referenced a field renamed by the cloud provider months earlier. After fixing, coverage rises to 6 of 8 tested (75%).
Synthetic case. Statuses come from the lab script; test results are invented.
Errors and limits
Coverage percentages hide depth: one rule per technique may cover only one way of doing it.
Exercises
- EX079 — Honest coverage. Why are untested rules excluded from coverage? Rubric: 3 = may silently fail; not proven; 2 = partial; 1 = included; 0 = none.
- EX080 — Lab coverage. Report the coverage figures and gaps. Rubric: 3 = 4/8 tested 50%, 2 untested, 2 none: T1021, T1048; 2 = partial; 1 = wrong; 0 = none.
- EX081 — Broken rule. Why was the email collection rule broken? Rubric: 3 = referenced field renamed by provider; 2 = partial; 1 = unknown; 0 = none.
Synthesis
Verify each rule with safe tests and count only tested rules as coverage.
Continue studying — chapter 27 of 40
← Chapter 26 (draft) · Course chapter index · Chapter 28 (draft) →