OpenCyber Academy
Doctoral research · Chapter 5 · English
Course code: OCA-199Chapter 05: OCA-199-C005

Chapter 5 — Main study and independent validation

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 05 published. Map, example, EX027.

flowchart TD
  N0["Learning outcomes"]
  N1["5.1 Design"]
  N2["5.2 Primary measure"]
  N3["Lab — intervals for accuracy"]
  N4["5.3 Reading the result"]
  N5["5.4 Harms"]
  N6["5.5 Independent validation"]
  N7["5.6 Time"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX027"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["5.1 Design"]
  N2["5.2 Primary measure"]
  N3["Lab — intervals for accuracy"]
  N4["5.3 Reading the result"]
  N5["5.4 Harms"]
  N6["5.5 Independent validation"]
  N7["5.6 Time"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX027"]

[Learning outcomes]
    |
    v
[5.1 Design]
    |
    v
[5.2 Primary measure]
    |
    v
[Lab — intervals for accuracy]
    |
    v
[5.3 Reading the result]
    |
    v
[5.4 Harms]
    |
    v
[5.5 Independent validation]
    |
    v
[5.6 Time]
    |
    v
[Worked example] --> [Exercise EX027]

Learning outcomes

You will (1) run a controlled crossover study; (2) estimate effects with intervals; (3) arrange independent validation of analysis; (4) report effects honestly, including harms.

5.1 Design

Eight analysts each triage 40 alerts: 20 baseline cards and 20 enriched cards, in counterbalanced order, with alert sets swapped between halves. Each of the 320 corpus alerts is triaged by exactly one analyst, so the study has 320 decisions, 160 per condition. Enriched cards add asset criticality, user role, and the count of similar alerts closed as benign in the last 30 days. Counterbalancing means half the analysts see baseline cards first and half see enriched cards first, so learning and fatigue affect both conditions equally. Swapping alert sets between halves ensures that differences come from the cards, not from easier alerts in one condition. The three enriched fields were chosen from the Contextual property and from what Northgate's analysts said they look up most often.

5.2 Primary measure

Accuracy: the share of dispositions matching ground truth. Pooled across analysts, each condition has 160 alert decisions (8 analysts × 20), and all of them enter the registered interval. Accuracy is scored against the corpus labels confirmed by the senior analysts. Decisions are logged automatically, including any change of disposition before submission, so the final decision is the one scored. Time-to-disposition is recorded in the same log but treated as secondary, as registered. An alert is scored correct only when the disposition matches the label exactly; escalating a benign alert counts as an error, just as closing an attack does, because both cost a SOC time or safety.

Lab — intervals for accuracy

"""Wilson 95% intervals for triage accuracy in two synthetic conditions."""
import math
def wilson(k, n, z=1.96):
    p = k / n
    d = 1 + z*z/n
    c = (p + z*z/(2*n)) / d
    h = z * math.sqrt(p*(1-p)/n + z*z/(4*n*n)) / d
    return c - h, c + h
for name, k, n in (("baseline", 118, 160), ("enriched", 139, 160)):
    lo, hi = wilson(k, n)
    print(f"{name:9} {k}/{n} = {k/n:.3f}  95% CI [{lo:.3f}, {hi:.3f}]")

Run with python3 lab_wilson.py. Verified output:

baseline  118/160 = 0.738  95% CI [0.664, 0.800]
enriched  139/160 = 0.869  95% CI [0.808, 0.913]

5.3 Reading the result

Baseline accuracy is 118/160 = 0.738 (95% CI 0.664–0.800); enriched is 139/160 = 0.869 (0.808–0.913). The intervals do not quite overlap (0.800 against 0.808), and the difference of 13 points is consistent with the hypothesis. Because the same analysts did both conditions, the registered paired analysis is the main test; the Wilson intervals describe each condition. They treat the 160 pooled decisions as independent, although each analyst contributes 20 of them, so they are likely somewhat too narrow if analysts differ in skill; this is why they are descriptive only. Nadia reports the interval for each condition and leaves the main inference to the registered paired test. The two intervals together show readers the scale of the change. Neither interval is used to claim significance on its own, since overlapping or separate intervals are not a formal test.

5.4 Harms

Of the 21 additional correct decisions, most were benign triggers correctly closed. But two true attacks were wrongly closed in the enriched condition, both carrying the note "similar alerts closed as benign: 14". History context can lull analysts. This is reported prominently. Nadia reports the two missed attacks in the abstract, not only in the results, because a reader deploying enriched cards needs to know about them. She examines both alerts in detail as part of her failure analysis, and the history field becomes the focus of the follow-up and the critique that followed.

5.5 Independent validation

A statistician at another university reruns the analysis from Nadia's raw logs and scripts without seeing her results. Her figures match to three decimals, which also checks that the scripts are complete. The statistician received the registered plan and the anonymised logs, not Nadia's tables, so agreement shows that the analysis follows from the plan and the data. The validation report is in the archive with her name and university. Prof. Grant arranged the statistician through Calder University, someone with no part in the study or in Nadia's supervision.

5.6 Time

Median time-to-disposition fell from 96 to 81 seconds; the time difference is secondary and labelled as such. Because time was registered as secondary, Nadia does not test it as a main hypothesis or build claims on it. She notes that faster decisions on enriched cards are consistent with analysts using context to settle benign triggers quickly, but the same speed could contribute to the two missed attacks. The time data are released with the logs for others to analyse. The time comparison is also reported per analyst, so readers can see whether the drop from 96 to 81 seconds came from a few people.

Deviations from the plan

One analyst missed a session through illness and completed it a week later. The registration allowed this; the deviation is reported with the date.

Worked example

Nadia's results paragraph: "Enriched cards raised accuracy from 0.738 to 0.869 (paired difference 0.131, 95% CI per registered analysis). Two true attacks were closed incorrectly in the enriched condition, both shown a high benign-history count; we treat this as a design hazard."

Synthetic case. Figures come from the lab script.

Errors and limits

Eight analysts from one SOC limit generalisation; analyst effects may dominate.

Exercises

Synthesis

Controlled designs, interval estimates, independent reanalysis and honest reporting of harms make a main study credible.

Continue studying — chapter 05 of 08

← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →

Worked answers

EX027. Every analyst triaged in both conditions, in counterbalanced order, with the alert sets exchanged between halves.

EX028. The asset's criticality, the user's role and how many similar alerts had been closed as benign in the past 30 days.

EX029. Baseline 0.738 (95% CI 0.664–0.800); enriched 0.869 (0.808–0.913).

EX030. Because each analyst contributed to both conditions, a paired comparison removes differences between analysts.

EX031. Two genuine attacks were dismissed in the enriched condition, each displaying a large benign-history count.

EX032. A statistician at another university repeated the analysis from raw logs and scripts without seeing Nadia's figures, and obtained the same numbers.