Chapter 4 — Baseline replication and pilot
Lesson map
Lesson 04 published. Map, example, EX021.
flowchart TD
N0["Learning outcomes"]
N1["4.1 Why replicate a baseline"]
N2["4.2 Ground-truth labels"]
N3["Lab — agreement on labels"]
N4["4.3 Reading the result"]
N5["4.4 The pilot"]
N6["4.5 What the pilot found"]
N7["4.6 Pilot data"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX021"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["4.1 Why replicate a baseline"]
N2["4.2 Ground-truth labels"]
N3["Lab — agreement on labels"]
N4["4.3 Reading the result"]
N5["4.4 The pilot"]
N6["4.5 What the pilot found"]
N7["4.6 Pilot data"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX021"]
[Learning outcomes]
|
v
[4.1 Why replicate a baseline]
|
v
[4.2 Ground-truth labels]
|
v
[Lab — agreement on labels]
|
v
[4.3 Reading the result]
|
v
[4.4 The pilot]
|
v
[4.5 What the pilot found]
|
v
[4.6 Pilot data]
|
v
[Worked example] --> [Exercise EX021]
Learning outcomes
You will (1) replicate a baseline before testing an intervention; (2) measure inter-rater reliability of ground-truth labels; (3) run a pilot to test procedures; (4) revise the protocol from pilot evidence.
4.1 Why replicate a baseline
Before claiming enrichment helps, Nadia must show her setup reproduces known behaviour: analysts in the baseline condition should show accuracy and benign-trigger confusion similar to what practitioners report. If the baseline is odd, the comparison is meaningless. Nadia's check is that baseline analysts confuse benign triggers with attacks at a rate similar to what practitioners describe. If baseline accuracy were near perfect, enrichment would have nothing to improve and the corpus would be too easy. Her pilot and baseline results are compared with the benign-trigger reports in the literature before the main analysis.
4.2 Ground-truth labels
The corpus labels are written by Nadia when she generates alerts, but "malicious" versus "benign" can be debatable for borderline cases. Two senior analysts independently label 20 sample alerts to test whether labels are clear. If the two senior analysts disagree often, the ground truth itself is unclear and accuracy cannot be scored fairly. The 20 alerts are drawn to include borderline cases, not only obvious ones, so the test is demanding. The senior analysts do not take part in the main study.
Lab — agreement on labels
"""Cohen's kappa for two analysts labelling the same 20 synthetic alerts."""
from collections import Counter
a = "MMBBBMBBBBMBBBBBMBBB" # M = malicious, B = benign (analyst A)
b = "MMBBBBBBBBMBBBMBMBBB" # analyst B
n = len(a)
po = sum(x == y for x, y in zip(a, b)) / n
ca, cb = Counter(a), Counter(b)
pe = sum(ca[k] * cb[k] for k in "MB") / n**2
kappa = (po - pe) / (1 - pe)
print(f"n={n} observed agreement={po:.2f} chance={pe:.3f} kappa={kappa:.2f}")
Run with python3 lab_kappa.py. Verified output:
n=20 observed agreement=0.90 chance=0.625 kappa=0.73
4.3 Reading the result
The two analysts agree on 18 of 20 alerts (0.90). Chance agreement, given how often each says "malicious", is 0.625, so Cohen's kappa is 0.73 — "substantial" on the Landis and Koch (1977) scale. The two disagreements are discussed; one label is corrected and one alert is rewritten to remove ambiguity. Kappa is lower than the raw agreement because both analysts called most alerts benign, which makes chance agreement high. The corrected label and the rewritten alert are recorded in the change log, including alert 17. With these fixes, Nadia treats the corpus labels as reliable enough to score the main study.
4.4 The pilot
Two volunteer analysts not in the main study triage 40 alerts, 20 in each condition. The pilot checks timing, instructions, the alert-card layout and logging. The pilot analysts use the same cards and instructions as the main study, so any problem they meet would also affect the main sessions. They are asked afterwards what confused them, and Nadia watches one session to see where they hesitate. Their logs are kept separately and are not part of any reported effect. Pilot analysts also time each session, which is how the 70-minute run for 40 alerts was found.
4.5 What the pilot found
Analysts did not notice the asset criticality field because it sat at the bottom of the card; logging missed disposition changes; 40 alerts took 70 minutes, causing fatigue. Nadia moves context fields near the top, fixes logging, and splits the 40 alerts into two sessions of 20 with a 15-minute break. Each change is listed in the v1.1 change log before the main study begins. The fatigue finding matters for validity: tired analysts make more errors late in a session, which could bias whichever condition comes second.
4.6 Pilot data
Pilot results are not used to estimate effects; they test procedure. This is stated in the registration. Using pilot data for estimates would mix procedure testing with outcome measurement, and the pilot cards differed from the final ones after the layout change. Keeping them apart means the main results come only from the registered design. Nadia still reports what the pilot found about procedure, because it explains why the final cards look as they do. The registered plan states that pilot logs are excluded from every reported estimate, so examiners can check this against the archive.
Worked example
Nadia's protocol change log: "v1.1 — context fields moved above the raw event; disposition-change logging added; sessions split into 2 × 20 alerts with a 15-minute break; alert 17 rewritten after label disagreement."
Synthetic case. Labels come from the lab script.
Errors and limits
Kappa depends on prevalence; with very unbalanced labels, high agreement can still give low kappa.
Exercises
- EX021 — Baseline purpose. Why replicate baseline behaviour first? Rubric: 3 = an odd baseline makes the comparison meaningless; 2 = partial; 1 = habit; 0 = none.
- EX022 — Kappa result. Report observed agreement, chance agreement and kappa. Rubric: 3 = 0.90, 0.625, 0.73; 2 = two; 1 = one; 0 = none.
- EX023 — Interpreting kappa. How is 0.73 described on the Landis and Koch scale? Rubric: 3 = substantial (0.61–0.80); 2 = substantial only; 1 = wrong; 0 = none.
- EX024 — Disagreements. What happened to the two disagreements? Rubric: 3 = discussed; one label corrected, one alert rewritten; 2 = partial; 1 = ignored; 0 = none.
- EX025 — Pilot findings. Name three pilot findings and their fixes. Rubric: 3 = field placement, logging, fatigue with fixes; 2 = two; 1 = one; 0 = none.
- EX026 — Kappa caveat. What caveat applies to kappa? Rubric: 3 = prevalence effect; unbalanced labels lower kappa despite agreement; 2 = partial; 1 = vague; 0 = none.
Synthesis
Replicate the baseline, check label reliability, pilot procedures and revise transparently before the main study.
Continue studying — chapter 04 of 08
← Chapter 03 (draft) · Course chapter index · Chapter 05 (draft) →