Chapter 29 — Authorized offensive security and red team: measurement
Lesson map
Lesson 29 published. Map, example, EX085.
flowchart TD
N0["Learning outcomes"]
N1["29.1 Coverage needs a plan"]
N2["29.2 Activity vs outcome"]
N3["29.3 One-sided error"]
N4["29.2b Sampling without lying"]
N5["29.3b Dashboards as arguments"]
N6["29.1b Cohort comparisons"]
N7["Practice note"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX085"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["29.1 Coverage needs a plan"]
N2["29.2 Activity vs outcome"]
N3["29.3 One-sided error"]
N4["29.2b Sampling without lying"]
N5["29.3b Dashboards as arguments"]
N6["29.1b Cohort comparisons"]
N7["Practice note"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX085"]
[Learning outcomes]
|
v
[29.1 Coverage needs a plan]
|
v
[29.2 Activity vs outcome]
|
v
[29.3 One-sided error]
|
v
[29.2b Sampling without lying]
|
v
[29.3b Dashboards as arguments]
|
v
[29.1b Cohort comparisons]
|
v
[Practice note]
|
v
[Worked example] --> [Exercise EX085]
Learning outcomes
You will (1) define coverage with a denominator; (2) separate activity metrics from outcome metrics; (3) publish a measurement that can be wrong in only one direction.
29.1 Coverage needs a plan
ATT&CK coverage is observed ∩ planned / |planned|, not observed / |all ATT&CK|. Without a plan, "coverage" is vanity. PulseMap requires a planned set per engagement.
"""ATT&CK-style coverage matrix over synthetic authorized techniques (names only)."""
# Synthetic engagement log: technique_id -> observed (True/False)
PLAN = ["T1046", "T1595", "T1082", "T1083", "T1059", "T1078", "T1021"]
OBSERVED = {"T1046", "T1595", "T1082", "T1083", "T1078"}
LABELS = {
"T1046": "Network Service Discovery",
"T1595": "Active Scanning",
"T1082": "System Information Discovery",
"T1083": "File and Directory Discovery",
"T1059": "Command and Scripting Interpreter",
"T1078": "Valid Accounts",
"T1021": "Remote Services",
}
covered = 0
for tid in PLAN:
hit = tid in OBSERVED
covered += hit
print(f"{tid} {LABELS[tid]:34} {'COVERED' if hit else 'PLANNED-NOT-SEEN'}")
print(f"coverage={covered}/{len(PLAN)} ({100*covered/len(PLAN):.0f}%)")
Expected output:
T1046 Network Service Discovery COVERED
T1595 Active Scanning COVERED
T1082 System Information Discovery COVERED
T1083 File and Directory Discovery COVERED
T1059 Command and Scripting Interpreter PLANNED-NOT-SEEN
T1078 Valid Accounts COVERED
T1021 Remote Services PLANNED-NOT-SEEN
coverage=5/7 (71%)
Planned sets make the denominator a promise, not a scavenger hunt through all of ATT&CK. Vanity percentages that divide by the whole matrix hide skipped techniques. PulseMap refuses to render coverage until the planned set is saved with the engagement id. Sam reviews the planned set with the client contact so both sides agree what incomplete means.
29.2 Activity vs outcome
Activity: tools launched, packets sent. Outcome: verified remediations, SLA breaches closed, out-of-scope probes. Steer on outcomes; use activity only for capacity planning.
Packets and tool launches fill capacity charts; they do not prove riders are safer. Sam keeps activity metrics on a separate dashboard labelled capacity only, so nobody confuses busyness with progress. When Riverstone's leadership asks how hard the team worked last month, Sam shows tool-hours on the capacity board and verified remediations on the outcome board, never a single chart that mixes the two.
29.3 One-sided error
Prefer metrics that, if wrong, under-claim success (e.g. count a fix only after re-test). Over-claiming security is the failure mode Albrecht and Jensen attack.
Synthetic case — Riverstone Transit. Coverage 5/7 (71%) with two planned-not-seen — the report said "incomplete planned coverage", not "71% safe".
Counting a fix only after re-test biases toward under-claiming, which is the safer lie if a lie must exist. At Riverstone a finding moves to closed in PulseMap only after Sam's re-test shows it gone from the same synthetic host, so the two planned-not-seen techniques stayed open rather than counted as safe. The synthetic report's wording, incomplete planned coverage, models the habit Sam wants in every PulseMap export.
29.4 Errors and limits
Denominators stolen from another engagement invalidate the metric.
Borrowing last quarter's denominator silently invents coverage. Each engagement carries its own planned set file, hashed into the dossier. Sam's CI fails a dashboard build that references a planned set from a different engagement id, and reviewers treat a missing engagement id on a widget as a defect equal to a wrong formula. A notebook that joins March and September planned sets for a paper must say so in its header and cannot feed the live PulseMap path that operators use during a window.
29.2b Sampling without lying
If you sample 50 of 500 hosts, say so and do not scale the finding count as if you visited all 500. Stratify by exposure when possible (Chapter 11 inventory). A honest sample with a confidence note beats a fake census.
Scaling sampled findings to the full estate is how small labs become fake censuses. Stratifying by exposure keeps rare high-impact hosts from vanishing in averages. A confidence note next to the sample size is mandatory in Sam's report template before the finding count is shown.
29.3b Dashboards as arguments
Every dashboard widget is a claim. Caption widgets with the formula and the engagement id. A widget without a formula is decoration and should be removed in review.
Formulas and engagement ids turn widgets into checkable claims. Decoration without formulas waste review time and imply precision they lack. Sam deletes uncaptioned widgets in the same pull request that would have added them, and he links each remaining caption to the notebook cell that computes it. When a designer proposed a green badge for "coverage health" without a planned set, Sam refused it in the same pull request and left the refusal note in the changelog for the next review.
29.1b Cohort comparisons
Comparing coverage across engagements requires the same planning rules. Publish the planning guide beside PulseMap so "71%" in March means the same kind of denominator as "71%" in September.
Same planning rules make March and September percentages comparable. A published planning guide beside PulseMap is the contract for that sameness. Sam versions the guide and records which version each engagement used, so a rule change cannot rewrite history by accident when someone reloads an old dashboard. The kickoff checklist requires a published guide pin; a blank pin or a pin to a draft branch blocks the start button the same way a missing ROE hash does.
Practice note
When leadership asks for a trend line, give them the same formula across engagements and mark methodology changes with vertical lines on the chart. A smooth upward coverage curve that hides a denominator change is a reporting failure. PulseMap exports include a methodology_id field so dishonest joins across incompatible plans are harder to automate.
Exercises
- EX085 — Formula. Write PulseMap's coverage formula and plug in the lab numbers. Rubric: 3 = 5/7; 2 = formula only; 1 = 5/all ATT&CK; 0 = none.
- EX086 — Metric pick. Choose outcome metric for remediation quality. Rubric: 3 = e.g. % critical fixed within SLA with re-test; 2 = activity metric; 1 = vague; 0 = none.
- EX087 — Under-claim. Give one design that under-claims success. Rubric: 3 = require re-test before closed; 2 = partial; 1 = vague; 0 = none.
Synthesis
Plan the denominator; measure outcomes; prefer under-claim.
Continue studying — chapter 29 of 50
← Chapter 28 (draft) · Course chapter index · Chapter 30 (draft) →