Chapter 4 — Replication of baselines and pilot protocol

Lesson map

Lesson 04 published. Map, example, EX021.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["4.1 Why replicate a baseline first"]
  N2["4.2 A reproduction procedure"]
  N3["4.3 Dataset shift and temporal leakage"]
  N4["4.4 Power and sample size"]
  N5["4.5 Pilot studies: feasibility, not effect"]
  N6["4.7 Making the computation reproducible"]
  N7["4.8 Practical activity: write the pilot report"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX021"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["4.1 Why replicate a baseline first"]
  N2["4.2 A reproduction procedure"]
  N3["4.3 Dataset shift and temporal leakage"]
  N4["4.4 Power and sample size"]
  N5["4.5 Pilot studies: feasibility, not effect"]
  N6["4.7 Making the computation reproducible"]
  N7["4.8 Practical activity: write the pilot report"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX021"]

[Learning outcomes and connection]
    |
    v
[4.1 Why replicate a baseline first]
    |
    v
[4.2 A reproduction procedure]
    |
    v
[4.3 Dataset shift and temporal leakage]
    |
    v
[4.4 Power and sample size]
    |
    v
[4.5 Pilot studies: feasibility, not effect]
    |
    v
[4.7 Making the computation reproducible]
    |
    v
[4.8 Practical activity: write the pilot report]
    |
    v
[Worked example] --> [Exercise EX021]

Learning outcomes and connection

By the end of this chapter the learner will be able to:

1. Attempt to reproduce a published baseline, document every discrepancy and diagnose its cause before claiming improvement.
2. Recognize dataset shift — covariate, label and concept shift — and temporal leakage in incident-response datasets and benchmarks.
3. Plan sample size from the smallest effect of interest, using a transparent calculation and a sensitivity range.
4. Design a pilot study whose purpose is feasibility, not effect estimation, with predeclared progression criteria.
5. Make computational work reproducible: versioned data, pinned environment, recorded seeds, hashes and a run log.

Chapter 3 proposed the Handoff Study and stated failure conditions F1 and F2, which depend on pilot results. This chapter tests the ground under the main study: are the baselines what we think they are, and does the procedure work? Chapter 5 runs the main study only if the pilot's progression criteria are met.

4.1 Why replicate a baseline first

A claim of improvement is a claim about a difference. If the baseline is wrong, the difference is meaningless. Baselines go wrong in incident-response research for ordinary reasons: a benchmark has several versions; an evaluation script differs from the one described; a split was random in the code but described as temporal; labels were corrected after publication; preprocessing silently dropped rows. Arp et al. (2022) list inappropriate baselines and data snooping among the pitfalls they found widespread in security papers using machine learning, and Pendlebury et al. (2019) showed that time splits which would be impossible in deployment inflated published malware-classification results.

Using the definitions adopted in Chapter 1 from the National Academies (2019), reproducing a baseline is a reproducibility exercise (same data, code and conditions) when the original artifacts are available, and becomes a replication when you must re-collect data. In either case, the goal is not to embarrass the original authors but to establish the reference point for your own claim.

4.2 A reproduction procedure

1. Freeze the target. Record the exact numbers, tables and conditions you intend to reproduce, with page or table references.
2. Obtain artifacts. Data (with version identifier and hash), code (with commit), environment, splits, seeds, preprocessing steps. List what is missing.
3. Run as published. Make no improvements yet. Record environment details and all outputs.
4. Compare. For each target number, record reproduced value, difference and whether it lies within a tolerance you set before running.
5. Diagnose discrepancies systematically: data version, split, preprocessing, metric definition (micro vs. macro averaging, label set), randomness, software versions, hardware nondeterminism.
6. Contact authors with a precise, courteous description if a discrepancy remains. Record the exchange.
7. Report the reproduction honestly, including what could not be reproduced, and choose your comparison point: the reproduced value under documented conditions, not the published number.

A reproduction log captures each step with timestamps and decisions. It belongs in the thesis archive (Chapter 8).

4.3 Dataset shift and temporal leakage

Incident data change over time: adversaries adapt, organizations deploy new tools, taxonomies are revised and analysts relabel. Three kinds of shift matter:

• Covariate shift: the distribution of inputs changes (new log sources, new alert types) while the relation between input and label is stable.
• Label shift: class proportions change (a phishing wave raises the share of phishing cases), changing base rates and thus positive predictive value (Chapter 1 §1.6.5).
• Concept shift (drift): the relation between inputs and labels changes (the same alert pattern now indicates a different activity, or a label definition changes).

Temporal leakage occurs when information from the future enters training or tuning: a random split that puts later incidents in training and earlier ones in testing; features computed with knowledge available only after the incident; or labels revised using hindsight. The deployment-faithful rule is simple: train only on what would have been available at the time of each prediction.

Human-study analogues exist. A scenario set calibrated on responders who later serve as participants leaks information; a codebook revised after seeing main-study decisions leaks outcome knowledge into measurement. Treat these with the same rule: freeze instruments before the data they will evaluate.

4.4 Power and sample size

Statistical power is the probability that a study detects an effect of a given size if it exists. Underpowered studies waste participants' time, produce unstable estimates and, when combined with selective publication, inflate published effect sizes (Chapter 1 §1.3.1). Cohen (1992) provided a widely used primer on power analysis for common tests.

For a within-subject comparison like the Handoff Study's (each participant experiences both SHA and Standard), a normal-approximation sample size for a two-sided paired test is

n ≈ ((z₁₋α/₂ + z₁₋β) · σ_d / δ)²

where δ is the smallest effect of interest (Chapter 3 §3.3), σ_d is the standard deviation of within-participant differences, α is the two-sided significance level and 1 − β is the power. The difficult input is σ_d: it must come from prior data or a pilot, and it is uncertain. The defensible practice is to report a sensitivity range — required n across plausible σ_d values — and to plan for the pessimistic end. With small samples a t-based calculation or simulation is more accurate than the normal approximation; the approximation is used here for transparency.

Precision can replace power as the planning target: plan n so that the expected confidence-interval half-width is smaller than the SEOI. This aligns with Chapter 3's goal of making a null result informative.

4.5 Pilot studies: feasibility, not effect

A pilot asks whether the main study can be done as designed, not whether the intervention works. Effect estimates from small pilots are too imprecise to justify or cancel a main study and invite selective decisions. Pilot objectives for the Handoff Study:

Lesson table; scroll horizontally to read all columns →

In plain terms: A pilot tests whether the study can be run, not whether the intervention works; each objective has a threshold, decided in advance, that says proceed, amend or stop.

The outcome of a pilot is one of three decisions: proceed (all criteria met), proceed with amendments (fixable problems, documented as protocol revisions before the main study) or stop or redesign (a premise failed). Pilot participants should normally not be reused in the main study, because they have learned the scenarios.

4.6 Worked example: a synthetic benchmark whose published baseline cannot initially be reproduced

OpenTriage-S and its paper are synthetic. A (fictional) paper introduces OpenTriage-S, a benchmark of 20 triage cases across nine months, and reports a "source-majority" baseline — predict the most common label seen for each alert source in training — with accuracy 0.833 on its test split. The Handoff Study candidate wants to use OpenTriage-S to check a triage component and must first reproduce that number.

Attempt 1. The candidate downloads the current release (labels v1.1), uses the test-case list printed in the paper's appendix and obtains 0.667. The discrepancy exceeds the preregistered tolerance of ±0.02.

Diagnosis. The repository's changelog shows that v1.1 corrected three labels (T05, T08 and T09) after community review. The paper used v1.0. Re-running on v1.0 reproduces 0.833 exactly. The baseline is reproducible — but only on labels now known to contain errors.

A second problem. Inspecting the published split reveals that the training set includes cases from months 7–9 while the test set includes cases from months 1–6: the split is not time-respecting. In particular, the source "vpn" is assigned the label "account" because "account" cases appear only in months 7–9 — information that would not have existed when early cases were triaged. Under a time-respecting split (train months 1–6, test months 7–9) with corrected labels, accuracy falls to 0.375, partly because the label "account" never appears in training at all — a case of concept shift that the published split hid.

The offline script below reproduces all three numbers and also computes paired-design sample sizes for the Handoff Study's SEOI under two variance assumptions.

"""Chapter 4 offline lab. OpenTriage-S is a synthetic benchmark invented for this book."""
from collections import Counter, defaultdict
from statistics import NormalDist

# case_id, month, source, label_v1_0, label_v1_1
CASES = [
    ("T01", 1, "mail", "phish", "phish"), ("T02", 1, "edr", "malware", "malware"),
    ("T03", 2, "mail", "phish", "phish"), ("T04", 2, "vpn", "benign", "benign"),
    ("T05", 3, "mail", "phish", "benign"), ("T06", 3, "edr", "malware", "malware"),
    ("T07", 4, "vpn", "benign", "benign"), ("T08", 4, "mail", "phish", "benign"),
    ("T09", 5, "edr", "malware", "benign"), ("T10", 5, "vpn", "benign", "benign"),
    ("T11", 6, "mail", "phish", "phish"), ("T12", 6, "edr", "malware", "malware"),
    ("T13", 7, "vpn", "account", "account"), ("T14", 7, "mail", "phish", "phish"),
    ("T15", 8, "vpn", "account", "account"), ("T16", 8, "edr", "malware", "malware"),
    ("T17", 9, "vpn", "account", "account"), ("T18", 9, "mail", "benign", "benign"),
    ("T19", 9, "vpn", "account", "account"), ("T20", 9, "edr", "malware", "malware"),
]
PUBLISHED_TEST = {"T02", "T04", "T08", "T11", "T16", "T20"}  # split listed in the (synthetic) paper

def fit(train, label_idx):
    by_src = defaultdict(Counter)
    for c in train:
        by_src[c[2]][c[label_idx]] += 1
    return {s: cnt.most_common(1)[0][0] for s, cnt in by_src.items()}

def accuracy(model, test, label_idx):
    return sum(model.get(c[2]) == c[label_idx] for c in test) / len(test)

def run(name, train, test, label_idx):
    m = fit(train, label_idx)
    print(f"{name:34s} n_test={len(test):2d} acc={accuracy(m, test, label_idx):.3f}")

def paired_n(delta, sd_diff, alpha=0.05, power=0.80):
    z = NormalDist().inv_cdf
    return ((z(1 - alpha / 2) + z(power)) * sd_diff / delta) ** 2

if __name__ == "__main__":
    pub_train = [c for c in CASES if c[0] not in PUBLISHED_TEST]
    pub_test = [c for c in CASES if c[0] in PUBLISHED_TEST]
    run("published split, labels v1.0", pub_train, pub_test, 3)
    run("published split, labels v1.1", pub_train, pub_test, 4)
    t_train = [c for c in CASES if c[1] <= 6]
    t_test = [c for c in CASES if c[1] >= 7]
    run("time split (m1-6 / m7-9), v1.1", t_train, t_test, 4)
    print("unseen test labels in time split:", sorted({c[4] for c in t_test} - {c[4] for c in t_train}))
    for sd in (1.0, 1.5):
        print(f"paired n for delta=0.5, sd_diff={sd}: {paired_n(0.5, sd):.1f}")

Expected output (verified with Python 3):

published split, labels v1.0       n_test= 6 acc=0.833
published split, labels v1.1       n_test= 6 acc=0.667
time split (m1-6 / m7-9), v1.1     n_test= 8 acc=0.375
unseen test labels in time split: ['account']
paired n for delta=0.5, sd_diff=1.0: 31.4
paired n for delta=0.5, sd_diff=1.5: 70.6

What the candidate reports. "We reproduced the published OpenTriage-S baseline (0.833) using label version 1.0 and the published split. With corrected labels (v1.1) the same baseline scores 0.667. The published split is not time-respecting; under a time-respecting split with v1.1 labels the baseline scores 0.375, and one test label is absent from training. We use the time-respecting v1.1 configuration as our reference and report the others for comparability." This paragraph is more valuable to the field than a claim of beating 0.833.

Sample size. With δ = 0.5 errors per scenario, α = 0.05 and 80% power, the approximation requires about 32 participants if σ_d = 1.0 and about 71 if σ_d = 1.5. The pilot's variance estimate will choose between these; the candidate plans recruitment for the pessimistic case and reports the sensitivity range in the protocol.

4.7 Making the computation reproducible

Minimum practice for every analysis in a thesis:

Lesson table; scroll horizontally to read all columns →

In plain terms: Reproducibility is a set of small habits — pinned versions, fixed seeds, recorded hashes, one command to rerun — each cheap at the start and expensive to retrofit.

4.8 Practical activity: write the pilot report

After a pilot, write a short report with: objectives and criteria (copied from the protocol), results for each criterion, decision (proceed / amend / stop), list of protocol amendments with version numbers, and the updated sample-size calculation. The report is dated before the main study begins, establishing that amendments were made before main-study data existed.

For the SHA study, the report would state whether pilot rater agreement on error type met the protocol target (failure condition F1) and whether free-text handoffs already carried ruled-out hypotheses and telemetry gaps (F2). Either failure leads to amend or stop, never to a quiet start of the main study.

4.9 Reproducibility as a spectrum, not a badge

Peng (2011) described reproducibility in computational science as a spectrum running from publication alone, through publication with code, code and data, and finally linked and executable code and data, with full replication as the gold standard beyond it. The spectrum framing is useful for incident-response research because the field's constraints (Chapter 1 §1.4) often make the top of the spectrum unreachable for real data. A thesis can still state where on the spectrum each artifact sits and why, which is more informative than a binary "available / not available" statement.

Sandve et al. (2013) offered ten simple rules for reproducible computational research. Several translate directly into the baseline-reproduction step of an incident-response thesis:

• Record how every result was produced. For a baseline, this includes the exact dataset version, the split, the preprocessing, the model configuration and the random seeds — not only the code repository URL.
• Avoid manual data manipulation steps. Hand-editing a label file "to fix obvious errors" is a manual step that nobody can reproduce. If labels must be corrected, write the correction as code with a reason for each change, as the Chapter 4 lab does with the label-version difference.
• Archive the exact versions of external programs. Security tooling changes quickly; a parser update can change extracted features.
• Version-control custom scripts.
• Record intermediate results in standardized formats where possible, so that a discrepancy can be localized to a stage.
• Note the underlying random seeds for analyses that include randomness.
• Store raw data behind plots, and generate hierarchical analysis output so that summary numbers can be traced to their inputs.
• Connect textual statements to underlying results, so that every number in the thesis can be traced to code output.
• Provide public access to scripts, runs and results where constraints allow.

The last rule is where incident data usually stops. The answer is not to abandon the others: a fully scripted pipeline on restricted data, with a synthetic fixture that exercises the same code, still lets a reviewer check the computation even if they cannot check the inputs.

4.10 Leakage in incident data: a working taxonomy

Kapoor and Narayanan (2023) reviewed published reports of data leakage across scientific fields using machine learning and argued that leakage is a widespread cause of overoptimistic, irreproducible results. They proposed a taxonomy distinguishing, among other things, lack of a clean separation of training and test data, use of features that would not be legitimately available at prediction time, and test data drawn from a distribution different from the one the scientific claim concerns, and they proposed "model info sheets" for documenting how leakage was ruled out. Their taxonomy maps onto incident data with some security-specific instances:

Lesson table; scroll horizontally to read all columns →

In plain terms: Leakage means the model saw, directly or indirectly, information it would not have when making a real decision. Each row is a common way this happens with incident data, and the last column is the check that catches it.

The "label from the future" row deserves emphasis because it is easy to miss and because it bears on decision quality. A triage decision judged against a label assigned weeks later, after forensic analysis, conflates two questions: was the decision right given what was known, and was the case ultimately malicious? The Handoff Study's decision-point design (Chapter 5 §5.1) separates them for human decisions; baseline replications of automated triage should do the same for model decisions.

4.11 Documenting a baseline: datasheets and model cards

When a baseline is reproduced, the reproduction itself becomes an artifact that others will rely on. Two documentation formats from machine-learning research are useful templates. Gebru et al. (2021) proposed datasheets for datasets, which document a dataset's motivation, composition, collection process, preprocessing, uses, distribution and maintenance. Mitchell et al. (2019) proposed model cards, which document a trained model's intended use, the factors relevant to its performance, its metrics, evaluation data, training data, quantitative analyses, ethical considerations, and caveats.

Adapted to an incident-response baseline, the minimum documentation answers:

1. Collection context. Which organization(s), sector, period, and detection pipeline produced the cases? What was not collected (e.g., cases closed without tickets)?
2. Labeling. Who assigned labels, with what guidance, at what point in the workflow, and how often were labels revised? What is the inter-rater agreement, if measured?
3. Known shifts. Which events during the collection period changed the data-generating process — tooling changes, staffing changes, policy changes, large campaigns?
4. Intended use and misuse. For which decisions is the baseline intended to serve as a comparison? For which would it mislead?
5. Performance by factor. Results broken down by month, source, and case type, not only aggregated.

The OpenTriage-S worked example in §4.6 is synthetic, but the reproduction log learners write for EX021 and EX026 should follow this structure.

4.12 What pilots can and cannot tell you

Section 4.5 stated that pilots establish feasibility rather than effect. The reason deserves a precise statement, because candidates under time pressure are tempted to use a pilot's effect estimate to set the main study's sample size.

A pilot with, say, eight participants yields an effect estimate with a very wide interval. If the main study is powered on that point estimate, two problems follow. When the pilot happened to produce a large estimate, the main study will be underpowered for the true, smaller effect. When the pilot happened to produce a small estimate, the study may be abandoned although a meaningful effect exists. In both cases the decision is driven by noise. The SEOI-based justification of §3.13 avoids this: the main study is sized to detect the smallest effect worth knowing about, regardless of what the pilot estimated.

What pilots can estimate usefully:

• Variability of the outcome (the standard deviation of within-participant differences), with caution, because small samples estimate variance imprecisely too; a conservative choice is to plan with a value toward the upper end of plausible estimates.
• Rater agreement for each scored construct, which determines whether the outcome measure is usable.
• Completion rates, timing and dropout, which determine recruitment targets.
• Protocol failures: facilitator hints, scenario ambiguities, materials that reveal condition, and technical breakdowns.
• Floor and ceiling effects: if almost everyone makes zero errors in a scenario, that scenario cannot show an effect.

A pilot report that states "the pilot showed a promising effect" is a warning sign. A pilot report that states "the pilot showed the procedure is feasible, rater agreement for comprehension errors met the predeclared threshold, one scenario showed a ceiling effect and was replaced" is what a committee wants to read.

4.13 Replicating the human baseline, not only the model

Incident-response theses frequently compare a new aid with "current practice." Current practice is itself a baseline that needs replication. In the Handoff Study, the Standard condition stands in for free-text handoffs as practiced. Three checks establish that it is a fair baseline:

1. Ecological check. Collect a small sample of real, de-identified free-text handoffs (with permission) and compare their length and content categories with Standard-condition handoffs written in the simulation. If simulation handoffs are far shorter or omit categories that real ones contain, the baseline is a straw man.
2. Effort check. Senders in the Standard condition should receive the same time and the same instructions to be complete as SHA senders. A baseline written hastily inflates the comparison.
3. Expert endorsement. Ask practitioners whether Standard-condition handoffs look like what they receive. Record disagreement.

The general principle, shared with the reproduction of computational baselines, is that a comparison is only as informative as the weakest condition is honest.

Common errors and limits

• Comparing against a published number without reproducing it under your conditions.
• "Fixing" a baseline during reproduction, so the comparison is no longer to the published method.
• Random splits for time-ordered incident data.
• Treating a pilot's effect estimate as evidence for or against the intervention.
• Choosing σ_d optimistically to make the required sample feasible.
• Reusing pilot participants in the main study.
• Limit: the normal approximation underestimates n for small samples; use a t-based calculation or simulation for final planning.

Exercises — chapter 4

EX021 — Reproduction log. Objective: document a reproduction attempt. Inputs: the script in §4.6. Constraints: run offline; set tolerance ±0.02 before running. Steps: (1) record target (0.833); (2) run with v1.1 labels first and record the discrepancy; (3) diagnose using the case data; (4) run with v1.0; (5) write the log entry with environment details. Expected evidence: reproduction log. Rubric: 3 = exact outputs, label-version diagnosis, environment recorded, tolerance predeclared; 2 = one element missing; 1 = no diagnosis; 0 = not run.

EX022 — Leakage audit. Objective: detect temporal leakage. Inputs: the OpenTriage-S cases and published split. Constraints: identify specific case IDs. Steps: list test cases that occur earlier than some training cases; show which learned rule depends on future data; state the deployment-faithful split. Expected evidence: audit table and explanation. Rubric: 3 = identifies vpn→account learned from months 7–9 and its effect on T04, proposes time split; 2 = identifies leakage but not mechanism; 1 = vague; 0 = none.

EX023 — Shift classification. Objective: distinguish shift types. Inputs: (a) a SOC adds cloud audit logs in month 7; (b) a phishing wave triples phishing share in month 8; (c) the CSIRT redefines "account compromise" to include token theft in month 7; (d) OpenTriage-S's "account" label appears only after month 6. Constraints: one type per item with justification; consequence for evaluation. Steps: (1) for each scenario, compare training and evaluation distributions of inputs and labels; (2) classify the shift (covariate, label, concept or none); (3) state the consequence for the baseline estimate. Expected evidence: four paragraphs. Rubric: 3 = (a) covariate, (b) label shift, (c) concept shift, (d) concept/label emergence with justification; 2 = one error; 1 = two errors; 0 = none.

EX024 — Sample-size sensitivity. Objective: plan n transparently. Inputs: the paired_n function. Constraints: offline. Steps: compute n for δ = 0.5 with σ_d = 1.2; for δ = 0.25 with σ_d = 1.0; and for δ = 0.5, σ_d = 1.0 at 90% power; write a 120-word recommendation. Expected evidence: values and recommendation. Rubric: 3 = values correct to one decimal, recommendation plans for pessimistic case and notes approximation limits; 2 = values correct, recommendation thin; 1 = one value wrong; 0 = none.

EX025 — Pilot criteria. Objective: design feasibility criteria. Inputs: the Handoff Study and §4.5. Constraints: at least six criteria; at least one that can stop the study. Steps: write objectives, measures, thresholds and decisions. Expected evidence: table. Rubric: 3 = feasibility-focused, thresholds predeclared, stop criterion linked to a premise; 2 = includes effect-based criterion; 1 = vague; 0 = none.

EX026 — Reproducibility checklist for your own analysis. Objective: make a computation auditable. Inputs: any analysis you have run (or the Chapter 1 lab). Constraints: follow §4.7. Steps: produce data hashes, environment record, commit or file hash, and a run-log entry. Expected evidence: the record. Rubric: 3 = another person could rerun and compare; 2 = one element missing; 1 = description only; 0 = none.

Chapter synthesis

Before claiming improvement, establish the reference point. Reproduce the baseline as published, diagnose discrepancies, and choose a deployment-faithful configuration as the real comparison. Plan sample size from the smallest effect of interest with an honest variance range, and pilot for feasibility with predeclared progression criteria. The Handoff Study now has a reproduced, corrected reference for its triage component and a pilot that determines whether F1 and F2 are triggered. Chapter 5 runs the main study.