OpenCyber Academy
Postgraduate · Chapter 6 · English
Course code: OCA-153Chapter 06: OCA-153-C006

Chapter 6 — Security operations and blue team: controlled evaluation

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 06 published. Map, example, EX026.

flowchart TD
  N0["Learning outcomes"]
  N1["6.1 The change"]
  N2["6.2 Evaluation design"]
  N3["6.3 Measure"]
  N4["Lab — bootstrap difference in medians"]
  N5["6.4 Reading the result"]
  N6["6.5 Accuracy"]
  N7["6.6 Confounding"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX026"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["6.1 The change"]
  N2["6.2 Evaluation design"]
  N3["6.3 Measure"]
  N4["Lab — bootstrap difference in medians"]
  N5["6.4 Reading the result"]
  N6["6.5 Accuracy"]
  N7["6.6 Confounding"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX026"]

[Learning outcomes]
    |
    v
[6.1 The change]
    |
    v
[6.2 Evaluation design]
    |
    v
[6.3 Measure]
    |
    v
[Lab — bootstrap difference in medians]
    |
    v
[6.4 Reading the result]
    |
    v
[6.5 Accuracy]
    |
    v
[6.6 Confounding]
    |
    v
[Worked example] --> [Exercise EX026]

Learning outcomes

You will (1) design a controlled evaluation of a SOC process change; (2) analyse skewed operational data with medians and bootstrap intervals; (3) guard against confounding; (4) report effects with uncertainty.

6.1 The change

Daniel's team writes a new triage playbook for claims-export alerts that starts with three automated enrichment lookups — assignment data, contractor status and export destination — before analyst review. The three lookups answer the questions analysts ask most often on claims-export alerts: whether the records were assigned, whether the contractor is still active, and whether the export went to an expected destination. Putting the answers in front of the analyst before review is meant to shorten triage without changing who decides. The evaluation then tests whether triage actually shortened for claims-export alerts, without changing who made the decision.

6.2 Evaluation design

For four weeks, alerts are assigned alternately to the old and new playbook by alert number, so busy and quiet periods fall into both groups. Analysts rotate between playbooks. Sixteen alerts fall in each group. Assigning by alert number removes the team's choice of which alerts get the new playbook, which could otherwise favour easy alerts. Rotation spreads analyst skill across both groups. The design was written before the first alert was assigned, and the sample of 32 alerts was fixed by the four-week window rather than chosen after looking at results.

6.3 Measure

Triage time in minutes from pickup to decision, plus decision accuracy checked by a senior analyst. Triage time is taken from case-system fields, so it does not depend on analysts' memory. The senior analyst checking accuracy reviews each decision against the evidence without knowing in advance which playbook was used, as far as the case notes allow. Measuring accuracy beside speed guards against a playbook that is faster only because it skips checks, which would show up as more wrong decisions in that group.

Lab — bootstrap difference in medians

"""Bootstrap interval for the difference in median triage minutes between two playbooks (synthetic)."""
import random, statistics
random.seed(42)
old = [18, 22, 25, 19, 30, 27, 21, 24, 35, 20, 26, 23, 29, 31, 22, 28]
new = [14, 17, 16, 21, 15, 19, 18, 13, 25, 16, 20, 17, 22, 15, 19, 18]
obs = statistics.median(old) - statistics.median(new)
diffs = []
for _ in range(5000):
    a = [random.choice(old) for _ in old]; b = [random.choice(new) for _ in new]
    diffs.append(statistics.median(a) - statistics.median(b))
diffs.sort()
print(f"median old={statistics.median(old)} new={statistics.median(new)} difference={obs}")
print(f"bootstrap 95% interval [{diffs[124]}, {diffs[4874]}]")

Run with python3 lab_bootstrap.py. Verified output:

median old=24.5 new=17.5 difference=7.0
bootstrap 95% interval [3.5, 11.0]

6.4 Reading the result

The median triage time fell from 24.5 to 17.5 minutes, a difference of 7.0. The bootstrap 95% interval for the difference is 3.5 to 11.0 minutes, which excludes zero. Medians suit skewed times; bootstrapping gives an interval without assuming a distribution. With only 16 alerts per group, the interval is wide: the true saving could be as small as 3.5 minutes or as large as 11. Daniel reports the interval with the point estimate so that the board does not read 7.0 minutes as a precise figure. The three-month recheck adds alerts and should narrow it.

6.5 Accuracy

The senior analyst found one wrong decision in each group, so faster triage did not cost accuracy in this sample. One error per group in 16 alerts is too few to show that accuracy is equal; it shows only that the sample gives no sign of a loss. Daniel's report therefore says equal accuracy in this sample and plans to recheck error rates after three months of routine use, when more alerts will have been handled under the new playbook and novelty will have worn off.

6.6 Confounding

Alternating assignment protects against time-of-day effects; rotation protects against one fast analyst using one playbook. Remaining risk: analysts knew which playbook they used and may have tried harder with the new one. Daniel states this risk in his report and plans a recheck after three months, when the new playbook has become routine and any novelty effect has faded. If the saving shrinks markedly by then, part of the original gain was probably effort rather than enrichment. Blinding analysts was not possible, because the enrichment results are visible on the screen, so the recheck is the best available control.

Practical significance

A 7-minute saving per alert across about 400 claims-export alerts a month frees roughly 47 analyst hours, enough to fund a weekly threat-hunting session.

Next step

After three months the team repeats the measurement on routine data.

Worked example

Daniel's report: "The new playbook reduced median triage time by 7.0 minutes (95% bootstrap interval 3.5–11.0) over 32 alerts, with equal accuracy. Analysts were not blind to condition; we will recheck after three months of routine use."

Synthetic case. Times come from the lab script.

Errors and limits

Thirty-two alerts give a wide interval; operational change may wear off or improve with familiarity.

Exercises

Synthesis

Controlled operational evaluations need fair assignment, robust statistics, accuracy checks and honest notes on confounding.

Continue studying — chapter 06 of 12

← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →

Worked answers

EX026. Alternately by alert number, so busy and quiet periods appeared in both groups.

EX027. Old 24.5, new 17.5 minutes; difference 7.0; 95% interval 3.5–11.0.

EX028. Triage times are skewed, and the bootstrap gives an interval without assuming a distribution.

EX029. No; the senior review found one wrong decision in each group.

EX030. Analysts knew which playbook they used and may have worked harder with the new one.