Chapter 4 — Authorized offensive security and red team: competing explanations
Lesson map
Lesson 04 published. Map, example, EX021.
flowchart TD
N0["Learning outcomes"]
N1["4.1 Rival explanations"]
N2["4.2 Designing discriminators"]
N3["4.3 Base rates in queues"]
N4["4.4 Empty scores"]
N5["4.5 Claims you will refuse"]
N6["4.5b Pre-mortem"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX021"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["4.1 Rival explanations"]
N2["4.2 Designing discriminators"]
N3["4.3 Base rates in queues"]
N4["4.4 Empty scores"]
N5["4.5 Claims you will refuse"]
N6["4.5b Pre-mortem"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX021"]
[Learning outcomes]
|
v
[4.1 Rival explanations]
|
v
[4.2 Designing discriminators]
|
v
[4.3 Base rates in queues]
|
v
[4.4 Empty scores]
|
v
[4.5 Claims you will refuse]
|
v
[4.5b Pre-mortem]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX021]
Learning outcomes
You will (1) list alternative explanations for a remediation-rate difference; (2) design checks that separate them; (3) use base rates when interpreting ticket alerts; (4) avoid empty security scores; and (5) pre-specify which explanations you can test.
4.1 Rival explanations
If tagged findings remediate faster, rivals include: tags make tickets clearer; tags correlate with higher severity by construction; remediators treat research tickets differently (Hawthorne); ATT&CK tags were applied only when the writer was more careful overall.
Each rival predicts something different, which is what makes them testable. If tags simply make tickets clearer, the gain should appear across severities. If severity drives the difference, balancing severity should remove it. If remediators react to being studied, both arms should improve together. If careful writers tag more, the tag audit and writing quality checks should show it.
4.2 Designing discriminators
Balance severity and asset class across arms with stratified randomisation. Blind remediators to the research hypothesis. Audit a sample of tags for correctness against ATT&CK definitions. Measure time-to-first-response as well as 30-day completion.
Each check targets a rival. Stratification answers severity imbalance; blinding answers the Hawthorne effect; the tag audit answers poor tagging; and time-to-first-response shows whether tagged tickets are picked up sooner as well as finished sooner. Noah writes these checks into his pre-registration, so the analysis cannot choose them after the data arrives.
4.3 Base rates in queues
Security queues are full of low-prevalence true issues. A noisy urgent flag creates mostly false urgency. The same arithmetic that governs detection precision at low prevalence governs ticket escalation policies.
The base-rate lab makes this concrete. With a 95% true-positive rate and a 5% false-positive rate, a flag applied where 1% of tickets are truly urgent has a precision of about 0.161, so most flagged tickets are not urgent. Remediators who learn this stop trusting the flag, which can slow the work it was meant to speed up.
4.4 Empty scores
Albrecht and Jensen (arXiv:2010.06377) argue that OSSTMM's rav is not a valid operational security quantity. Your dissertation should not invent a new single score; compare concrete remediation outcomes under stated conditions.
For Noah, the concrete outcome is completion within 30 days, reported by arm with an interval. Readers can check it against the ticket export. A composite score would combine things that cannot be checked separately, and examiners would ask what a change of one point means. Noah's tables report outcomes with their counts.
4.5 Claims you will refuse
You will not claim that faster remediation means fewer breaches or that ATT&CK coverage equals safety. Say what ticket data can support — and stop there.
"""Lab B: base rates and alarm precision (Bayes' rule).
An urgent flag with given true- and false-positive rates is applied
to ticket queues with different shares of truly urgent issues."""
for tpr, fpr in ((0.95, 0.05), (0.95, 0.01), (0.80, 0.001)):
for prev in (0.10, 0.01, 0.001):
tp = tpr * prev; fp = fpr * (1 - prev)
prec = tp / (tp + fp)
per_10k = round(10000 * (tp + fp))
print(f"tpr={tpr:.2f} fpr={fpr:.3f} prevalence={prev:.3f} precision={prec:.3f} alerts_per_10k={per_10k}")
Expected output:
tpr=0.95 fpr=0.050 prevalence=0.100 precision=0.679 alerts_per_10k=1400
tpr=0.95 fpr=0.050 prevalence=0.010 precision=0.161 alerts_per_10k=590
tpr=0.95 fpr=0.050 prevalence=0.001 precision=0.019 alerts_per_10k=509
tpr=0.95 fpr=0.010 prevalence=0.100 precision=0.913 alerts_per_10k=1040
tpr=0.95 fpr=0.010 prevalence=0.010 precision=0.490 alerts_per_10k=194
tpr=0.95 fpr=0.010 prevalence=0.001 precision=0.087 alerts_per_10k=109
tpr=0.80 fpr=0.001 prevalence=0.100 precision=0.989 alerts_per_10k=809
tpr=0.80 fpr=0.001 prevalence=0.010 precision=0.890 alerts_per_10k=90
tpr=0.80 fpr=0.001 prevalence=0.001 precision=0.445 alerts_per_10k=18
Write the refused claims into the limitations chapter in plain words. For Noah, they include breach reduction, matrix coverage as safety and any claim about live enterprise programmes such as CBEST or TIBER-EU. Naming them early prevents the discussion chapter, or a manager reading it, from drifting past the evidence. Keep the list short and specific.
4.5b Pre-mortem
Before the main study, write a pre-mortem: "The study failed because…" listing severity imbalance, tag errors, UI blindness, staff turnover and authorization expiry. Assign a detector and a mitigation to each line. Pre-mortems catch issues cheaply.
Noah's detector for tag errors is the tag audit; his mitigation is second-person review. For authorization expiry, he notes the end date and asks for renewal a month before it. Write the pre-mortem as a short table and review it with the supervisor before the instrument lock. Severity imbalance is detected by checking arm balance after randomisation; staff turnover by tracking which remediators leave during collection.
4.6 Worked example (synthetic CBM case)
Noah pre-specifies rivals: severity imbalance, tag quality, Hawthorne effects. The base-rate lab shows that at 1% prevalence and 5% false positives, precision is about 0.161 — a warning against promiscuous urgency labels. He will report tag-audit agreement alongside the primary result.
Pre-specifying the rivals means Noah cannot add or drop them once results arrive. The precision figure supports his decision not to add urgent flags to research tickets, since most would be false at that prevalence. Reporting tag-audit agreement beside the primary result lets readers judge how much of the effect depends on correct tags.
4.7 Errors and limits
- Claiming breach reduction.
- Ignoring tag quality.
- Single composite security improvement scores.
Each error overstates what the study shows. Ticket data cannot measure breaches, so a claim of breach reduction goes beyond the evidence. Ignoring tag quality hides measurement error in the intervention itself. A single composite improvement score combines things that cannot be checked separately, the error Albrecht and Jensen describe. Check the discussion chapter for all three before submission. Noah's pre-registration and limitations chapter address each of them in writing.
Exercises
- EX021 — Rivals — List four rival explanations. Rubric: 3 = four apt; 2 = three; 1 = one; 0 = none.
- EX022 — Balance — How will you balance severity across arms? Rubric: 3 = stratification or constrained randomisation; 2 = partial; 1 = ignore; 0 = none.
- EX023 — Base rate — Using the lab, explain urgency-flag risk at 1% prevalence / 5% FPR. Rubric: 3 = precision ~0.16, alert burden; 2 = numbers; 1 = vague; 0 = wrong.
- EX024 — OSSTMM — Why avoid a rav-like score in this dissertation? Rubric: 3 = validity critique, prefer concrete outcomes; 2 = partial; 1 = vague; 0 = none.
- EX025 — Blind — How can remediators be blinded? Rubric: 3 = hide hypothesis/format purpose; 2 = partial; 1 = skip blinding; 0 = none.
- EX026 — Limits of claim — Write a claim you will not make. Rubric: 3 = breach/risk overclaim refused with reason; 2 = partial; 1 = still overclaims; 0 = none.
Synthesis
Competing explanations keep you honest: design checks for rivals and refuse claims your data cannot support.
Continue studying — chapter 04 of 08
← Chapter 03 (draft) · Course chapter index · Chapter 05 (draft) →