◇ OpenCyber Academy
Doctoral research · Chapter 6 · English

Chapter 6 — Independent validation across contexts

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 6 — Independent validation across contexts

Lesson map

Lesson 06 published. Map, example, EX033.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["6.1 Why incident-response results travel poorly"]
  N2["6.2 Kinds of replication"]
  N3["6.3 Heterogeneity"]
  N4["6.4 Boundary conditions as findings"]
  N5["6.5 Designing the second-context study"]
  N6["6.7 Practical activity: transportability memo"]
  N7["6.8 The generalizability problem, stated precisely"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX033"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["6.1 Why incident-response results travel poorly"]
  N2["6.2 Kinds of replication"]
  N3["6.3 Heterogeneity"]
  N4["6.4 Boundary conditions as findings"]
  N5["6.5 Designing the second-context study"]
  N6["6.7 Practical activity: transportability memo"]
  N7["6.8 The generalizability problem, stated precisely"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX033"]

[Learning outcomes and connection]
    |
    v
[6.1 Why incident-response results travel poorly]
    |
    v
[6.2 Kinds of replication]
    |
    v
[6.3 Heterogeneity]
    |
    v
[6.4 Boundary conditions as findings]
    |
    v
[6.5 Designing the second-context study]
    |
    v
[6.7 Practical activity: transportability memo]
    |
    v
[6.8 The generalizability problem, stated precisely]
    |
    v
[Worked example] --> [Exercise EX033]

Learning outcomes and connection

By the end of this chapter the learner will be able to:

  1. Explain external validity and transportability for incident-response findings, and identify the contextual features that plausibly modify an effect.
  2. Distinguish direct (close) replication from conceptual replication and choose between them for a stated purpose.
  3. Quantify heterogeneity between contexts and interpret it cautiously when few contexts exist.
  4. State boundary conditions — where a method stops working — as findings rather than failures.
  5. Design a second-context study that tests a predicted boundary rather than merely repeating the first study.

Chapter 5 found, in one synthetic organization, that SHA reduced comprehension and projection errors with uncertain practical importance. A single context cannot show whether the result depends on that organization's staffing, telemetry, culture or scenarios. This chapter tests the claim in a second context. Chapter 7 will then expose the combined evidence to critique.

6.1 Why incident-response results travel poorly

Chapter 1 (§1.1) noted that incidents are unrepeatable and sociotechnical. The consequence for generalization is that many features of a context can modify an effect:

Contextual feature Plausible modification of the SHA effect
Telemetry completeness Fewer gaps → less benefit from the T-gaps field.
Staffing model Follow-the-sun handoffs across regions → more handoffs, more language and time-zone barriers → possibly more benefit.
Responder experience Veterans may already convey ruled-out hypotheses → less benefit.
Existing handoff culture Strong verbal handoff norms → SHA redundant or disruptive.
Incident mix Routine incidents need little handoff context.
Tooling Case-management tools with structured fields → Standard already partly structured.
Regulatory pressure Documentation requirements may make SHA a compliance task, changing how it is filled in (the sociotechnical account in Chapter 2 §2.3.3).

In plain terms: Each contextual difference between organizations is a reason the effect might be larger, smaller or absent elsewhere; the table turns those reasons into testable moderators.

External validity asks whether a result holds for other populations, settings, treatments and outcomes. Transportability asks a sharper question: given what we know about how the target context differs, what effect should we expect there? Transportability reasoning requires a mechanism (Chapter 3) because only a mechanism tells us which differences matter.

6.2 Kinds of replication

The National Academies (2019) define replicability as obtaining consistent results across studies aimed at the same scientific question, each with its own data, and note that a rigorous study may still fail to replicate. Within that definition, it helps to distinguish:

For doctoral work, a boundary test is often the most informative second study, because it can strengthen the mechanism whichever way it comes out — provided the prediction was registered before the data were collected. The large collaborative replication effort reported by the Open Science Collaboration (2015), which attempted to replicate 100 published psychology studies, illustrates both the value of systematic replication and the debate over what counts as a successful replication; that debate is a reason to define the replication criterion in advance.

6.3 Heterogeneity

When an effect is estimated in several contexts, the estimates differ for two reasons: sampling error and real differences. Meta-analysis separates them. Under a fixed-effect model all contexts share one true effect; under a random-effects model true effects vary around a mean. Cochran's Q tests whether observed dispersion exceeds what sampling error would produce, and I², proposed by Higgins and Thompson (2002) and popularized in Higgins et al. (2003), describes the proportion of total variation attributable to heterogeneity rather than chance.

With only two or three contexts, these statistics are very imprecise: Q has little power, I² has a wide uncertainty interval, and random-effects variance estimates are unstable. A doctoral thesis with two contexts should therefore treat heterogeneity statistics as descriptive, rely primarily on the registered boundary prediction and the within-context moderator analysis, and avoid reporting a pooled effect as if it described any real context when heterogeneity is evident.

6.4 Boundary conditions as findings

A boundary condition is a value of a contextual feature beyond which the effect weakens, disappears or reverses. In incident-response practice, boundary conditions are highly actionable: they tell an organization whether to adopt a method. "SHA helps when telemetry is sparse but adds little where it is dense" is more useful than "SHA helps on average."

A boundary claim requires evidence of the same quality as a main effect. In particular, a boundary observed only across contexts is confounded by every other difference between those contexts. Stronger evidence comes from varying the feature within a context — for example, presenting scenarios with high and low telemetry gaps to the same participants — so that staffing, culture and tooling are held constant.

6.5 Designing the second-context study

The Handoff Study's second context is the synthetic Kestrel Freight Logistics SOC (KFL-SOC): a 24×7 tiered SOC with follow-the-sun handoffs between two regions, mature EDR coverage and a case-management tool with some structured fields. Compared with NMS-CSIRT, it differs on telemetry (dense vs. sparse), staffing (tiered, multi-region vs. small, single-site) and tooling.

Registered predictions (before KFL data collection):

Design choices: same SHA, codebook and rater training (direct replication of procedure); new scenarios adapted to logistics operations but with a crossed factor of telemetry-gap level (conceptual replication plus boundary test); raters from outside both organizations; preregistration of P6.1–P6.3 and of the analysis.

6.6 Worked example: a second synthetic service context with different staffing and telemetry quality

All values are synthetic. The script below takes the NMS effect from Chapter 5 (mean 1.417, standard error 1.165/√12 ≈ 0.336) and a KFL effect, pools them with inverse-variance weights, computes Q and I², and tests the KFL within-context gap interaction. Intervals use the normal approximation (±1.96 SE), so the NMS interval differs slightly from the t-interval in Chapter 5.

"""Chapter 6 offline lab: combine two synthetic contexts and test heterogeneity."""
from math import sqrt

# context: (mean reduction in comprehension+projection errors per 3 scenarios, standard error)
CONTEXTS = {
    "NMS-CSIRT (sparse telemetry)": (1.417, 0.336),
    "KFL-SOC (dense telemetry)":    (0.350, 0.250),
}
# KFL-SOC split by scenario telemetry-gap level: (mean reduction, SE)
KFL_BY_GAP = {"high-gap scenarios": (0.900, 0.400), "low-gap scenarios": (0.050, 0.300)}

def fixed_effect(effects):
    w = {k: 1 / se ** 2 for k, (m, se) in effects.items()}
    pooled = sum(w[k] * effects[k][0] for k in effects) / sum(w.values())
    se_pooled = sqrt(1 / sum(w.values()))
    q = sum(w[k] * (effects[k][0] - pooled) ** 2 for k in effects)
    df = len(effects) - 1
    i2 = max(0.0, (q - df) / q) if q > 0 else 0.0
    return pooled, se_pooled, q, i2

def diff(a, b):
    d = a[0] - b[0]
    se = sqrt(a[1] ** 2 + b[1] ** 2)
    return d, se, d / se

if __name__ == "__main__":
    for k, (m, se) in CONTEXTS.items():
        print(f"{k:30s} effect={m:.3f} 95%CI=({m - 1.96*se:.3f}, {m + 1.96*se:.3f})")
    pooled, se, q, i2 = fixed_effect(CONTEXTS)
    print(f"fixed-effect pooled={pooled:.3f} 95%CI=({pooled - 1.96*se:.3f}, {pooled + 1.96*se:.3f}) Q={q:.2f} I2={i2:.2f}")
    d, se_d, z = diff(*CONTEXTS.values())
    print(f"context difference={d:.3f} SE={se_d:.3f} z={z:.2f}")
    d, se_d, z = diff(KFL_BY_GAP["high-gap scenarios"], KFL_BY_GAP["low-gap scenarios"])
    print(f"KFL gap interaction={d:.3f} SE={se_d:.3f} z={z:.2f}")

Expected output (verified with Python 3):

NMS-CSIRT (sparse telemetry)   effect=1.417 95%CI=(0.758, 2.076)
KFL-SOC (dense telemetry)      effect=0.350 95%CI=(-0.140, 0.840)
fixed-effect pooled=0.730 95%CI=(0.337, 1.123) Q=6.49 I2=0.85
context difference=1.067 SE=0.419 z=2.55
KFL gap interaction=0.850 SE=0.500 z=1.70

Interpretation.

6.7 Practical activity: transportability memo

For a third organization of your choice (real or synthetic, described only at a level that exposes no confidential information), write a memo that: lists its values on each feature in §6.1; predicts the SHA effect using the mechanism and the two-context evidence; states the uncertainty; and proposes the smallest study that would check the prediction there. The memo is a reasoning exercise, not a recommendation to adopt.

A memo for a small library service such as Harbor Library would first have to say whether its responders hand off between shifts at all, since the mechanism works through handoffs.

6.8 The generalizability problem, stated precisely

Yarkoni (2022) argued that much of psychology suffers a "generalizability crisis": verbal claims are stated at a level of generality far beyond what the statistical models support, because those models treat only participants as a random sample while treating stimuli, tasks, settings and experimenters as fixed. If the verbal claim is about incident handoffs in general, then scenarios, organizations and facilitators are also samples from larger populations, and their variability belongs in the uncertainty of the estimate.

For incident-response research this critique has teeth. A study using four scenarios written by one researcher, run in one organization, can at most support claims about those scenarios in that organization unless the variation across scenarios and organizations is modeled or the claim is narrowed. Two practical consequences follow:

  1. Model scenarios as random effects (Chapter 5 §5.14) and, if there are several sites, sites as well. The resulting intervals will be wider — honestly so.
  2. Match the verbal claim to the sampled variation. If only one scenario author wrote all scenarios, say so, and consider recruiting a second author whose scenarios serve as a replication of stimulus sampling.

Simons, Shoda and Lindsay (2017) proposed that empirical papers include a Constraints on Generality (COG) statement specifying the target populations of participants, materials, procedures and contexts to which the authors expect the findings to generalize, and the reasons. A COG statement for the Handoff Study might read:

We expect the SHA's reduction in comprehension and projection errors to generalize to shift handoffs among responders with at least one year of operational experience, in incidents where investigation continues across shifts and telemetry is incomplete. We have no evidence about handoffs between organizations, handoffs under active legal constraint, or teams whose working language differs from the template's. We expect smaller or absent effects where telemetry is complete and incidents are routine, because the mechanism depends on transmitting uncertainty that such incidents lack.

The statement is falsifiable: a replication within the stated constraints that finds no effect counts against the claim, while a failure outside them marks a boundary rather than a refutation.

6.9 Transportability as a formal question

Pearl and Bareinboim (2014) treated external validity as a problem of transportability: when can a causal effect estimated in one population be used to infer the effect in another, given knowledge about how the populations differ? They represent differences as explicit features of a causal diagram and derive conditions under which the target effect can be computed from source experiments plus observational information in the target. The full formal apparatus exceeds this book; three of its ideas are directly usable.

The transportability memo in §6.7 is an informal application: features mapped, mechanism reasoned through, a minimal check proposed. The formal framework explains why the memo must distinguish "different mix" from "different mechanism."

6.10 Random-effects synthesis across several sites

With two contexts, Chapter 6 compared estimates directly. With more sites, a random-effects meta-analysis summarizes them. DerSimonian and Laird (1986) proposed a widely used method for meta-analysis in clinical trials that estimates the between-study variance (τ²) from the heterogeneity statistic Q and incorporates it into the pooled estimate's weights. The heterogeneity measures Q and I² were introduced in §6.3 with Higgins and Thompson (2002) and Higgins et al. (2003).

Two quantities answer different questions:

For practitioners deciding whether to adopt the SHA, the prediction interval is usually the more relevant number. The following lab uses a synthetic five-site extension of the Handoff Study.

# Synthetic five-site extension of the Handoff Study (illustrative only).
# site: (mean reduction in comprehension+projection errors, standard error)
import math
sites = {
    "NMS-CSIRT": (1.10, 0.25),
    "KFL-SOC":   (0.45, 0.30),
    "Site-C":    (0.90, 0.35),
    "Site-D":    (0.20, 0.40),
    "Site-E":    (1.30, 0.45),
}
y = [v[0] for v in sites.values()]
v = [v[1] ** 2 for v in sites.values()]
w = [1 / vi for vi in v]
fixed = sum(wi * yi for wi, yi in zip(w, y)) / sum(w)
Q = sum(wi * (yi - fixed) ** 2 for wi, yi in zip(w, y))
k = len(y)
C = sum(w) - sum(wi ** 2 for wi in w) / sum(w)
tau2 = max(0.0, (Q - (k - 1)) / C)
I2 = max(0.0, (Q - (k - 1)) / Q) * 100 if Q > 0 else 0.0
wr = [1 / (vi + tau2) for vi in v]
rand = sum(wi * yi for wi, yi in zip(wr, y)) / sum(wr)
se_r = math.sqrt(1 / sum(wr))
t_crit = 3.182  # t(0.975, df = k - 2 = 3)
pi_half = t_crit * math.sqrt(tau2 + se_r ** 2)
print(f"fixed-effect mean: {fixed:.3f}")
print(f"Q = {Q:.3f} on {k-1} df; tau^2 = {tau2:.4f}; I^2 = {I2:.1f}%")
print(f"random-effects mean: {rand:.3f} (SE {se_r:.3f})")
print(f"95% CI: [{rand-1.96*se_r:.3f}, {rand+1.96*se_r:.3f}]")
print(f"95% prediction interval: [{rand-pi_half:.3f}, {rand+pi_half:.3f}]")

Expected output:

fixed-effect mean: 0.810
Q = 6.363 on 4 df; tau^2 = 0.0666; I^2 = 37.1%
random-effects mean: 0.794 (SE 0.191)
95% CI: [0.420, 1.167]
95% prediction interval: [-0.227, 1.814]

Reading the output. The average effect across these five synthetic sites is clearly positive (random-effects mean 0.794, 95% CI 0.420 to 1.167). Moderate heterogeneity (I² ≈ 37%) means sites differ by more than sampling error alone. The prediction interval for a new site runs from −0.227 to 1.814: a new organization similar to these could plausibly see anything from a small worsening to a large improvement. The average is a confident claim; the prediction for any one site is not. With only five sites, τ² itself is estimated imprecisely, so even the prediction interval is approximate; the thesis should say so.

In plain terms: On average across sites the handoff template helps, and we are fairly sure of that. But for any single new organization, the plausible range includes "no benefit at all," because sites genuinely differ. Adoption decisions should look at the second range, not the first.

6.11 Organizational and cross-organizational context as moderators

Section 2.10 described research on response teams and on crisis coordination. For validation across contexts, these literatures suggest moderators that go beyond staffing and telemetry:

Each moderator should enter the transportability memo with a predicted direction derived from the mechanism, and the second-context study should measure at least the ones that differ between the two contexts.

6.12 Ecological validity: what simulation leaves out

A simulation can be made realistic in content and still differ from practice in ways that matter for the mechanism:

None of these invalidates the simulation result. Each is a specific reason the effect could differ in the field, and each suggests a field measure: for example, sampling real handoffs before and after SHA adoption and scoring them for transmitted hypotheses, ruled-out items and gaps — a mechanism measure that does not require judging real decisions. Such a study would test the first link of the causal chain in practice, which is a meaningful contribution even without an outcome measure.

Common errors and limits

Exercises — chapter 6

EX033 — Modifier analysis. Objective: identify contextual modifiers via mechanism. Inputs: the SHA mechanism (Chapter 3 §3.2) and §6.1. Constraints: at least six features; each tied to a link in the mechanism. Steps: list features, predicted direction, mechanism link and how to measure each. Expected evidence: table. Rubric: 3 = six or more features with mechanism links and measures; 2 = links weak; 1 = generic list; 0 = none.

EX034 — Replication type choice. Objective: match replication design to purpose. Inputs: three purposes: (a) check whether Chapter 5's result was a chance finding; (b) check whether results depend on the specific Harbor/CTN/LLR-seeded scenarios; (c) test whether telemetry density limits SHA. Constraints: justify each choice and name one design element. Steps: (1) list what will be held constant and what will vary; (2) choose direct, conceptual or extension replication; (3) justify the choice against the question Chapter 6 must answer. Expected evidence: three short paragraphs. Rubric: 3 = direct, conceptual, boundary test with sound elements; 2 = one mismatch; 1 = two; 0 = none.

EX035 — Heterogeneity computation. Objective: compute and interpret heterogeneity. Inputs: the script in §6.6. Constraints: offline. Steps: (1) run and confirm; (2) change KFL to (1.100, 0.250) and rerun; (3) interpret both Q/I² values with the caveat about two contexts. Expected evidence: outputs and paragraph. Rubric: 3 = correct values and caveated interpretation; 2 = values correct, no caveat; 1 = one value wrong; 0 = none.

EX036 — Across vs. within evidence. Objective: weigh confounded and unconfounded moderator evidence. Inputs: §6.6 results. Constraints: 200–300 words. Steps: explain why the within-KFL interaction is more diagnostic despite lower precision; propose a design that strengthens it. Expected evidence: paragraph. Rubric: 3 = explains confounding of across-context comparison, values precision appropriately, proposes stronger within design; 2 = partial; 1 = prefers z-value only; 0 = none.

EX037 — Revised claim with boundaries. Objective: write a boundary-aware claim. Inputs: Chapters 5–6 results. Constraints: ≤ 120 words; include scope, effect, boundary and uncertainty. Steps: (1) state the population and setting; (2) state the effect with its interval; (3) name the boundary condition and the evidence for it; (4) cut words until every remaining word carries scope or uncertainty. Expected evidence: claim. Rubric: 3 = scoped, calibrated, boundary as plausible not proven; 2 = one overstatement; 1 = pooled average as claim; 0 = none.

EX038 — Transportability memo. Objective: apply §6.7. Inputs: a third context description. Constraints: 400–600 words. Steps: follow §6.7. Expected evidence: memo. Rubric: 3 = features mapped, prediction reasoned through mechanism with uncertainty, minimal check proposed; 2 = prediction without uncertainty; 1 = adoption recommendation without evidence; 0 = none.

Chapter synthesis

A result from one incident-response context is a hypothesis about others. Testing it in a second context is most informative when the mechanism predicts where the effect should change and the prediction is registered before data. In the synthetic Handoff Study, the effect shrank in a dense-telemetry context and, within that context, was concentrated in high-gap scenarios — consistent with telemetry completeness as a boundary condition, though two contexts cannot establish it. Chapter 7 submits the combined claims to adversarial critique.

Continue studying — chapter 06 of 08

← Chapter 05 · Course chapter index · Chapter 07 →

Continue to the next authored lesson.

Worked answers

EX033 — Model modifier table.

Feature Predicted direction Mechanism link Measure
Telemetry completeness More gaps → larger effect T-gaps prevents absence-of-evidence errors Source coverage per asset class; gap count per scenario
Handoff frequency More handoffs → larger cumulative effect Context loss per transfer Handoffs per incident from case records
Responder experience More experience → smaller effect Veterans already transmit ruled-out hypotheses Tenure; pilot content analysis of free-text handoffs
Existing structured tooling More structure → smaller effect Standard already carries fields Field inventory of case tool
Language/time-zone distance Greater distance → larger effect Explicit Expect/Authority fields reduce ambiguity Region pairs in handoffs
Compliance pressure on documentation Higher pressure → possibly smaller effect (box-ticking) Field quality degrades Content-quality rating of SHA entries
Incident novelty More novel → larger effect Projection depends on transmitted expectations Scenario novelty rating

EX034 — Model replication choices. (a) Direct replication: same protocol, materials and scenarios with a new sample from a comparable population; key element — identical codebook and analysis, preregistered criterion for consistency (e.g., new interval overlaps the original and excludes zero). (b) Conceptual replication: new scenarios not seeded from Harbor/CTN/LLR, built by different designers, same hypotheses and codebook; key element — independent scenario authorship. (c) Boundary test: manipulate telemetry density as a crossed factor within participants, with a registered interaction prediction; key element — gap level varied within context to avoid confounding.

EX035 — Model heterogeneity. Original: pooled 0.730 (0.337, 1.123), Q = 6.49, I² = 0.85. Variant with KFL = (1.100, 0.250): pooled 1.213 (95% CI ≈ 0.820 to 1.606), Q = 0.57, I² = 0.00; context difference 0.317 (SE 0.419, z ≈ 0.76). Interpretation: in the original, the two estimates differ more than sampling error predicts, so a pooled value misdescribes both contexts. In the variant, there is no evidence of heterogeneity — but with two contexts Q has little power and I² = 0 does not demonstrate homogeneity; it means the data cannot distinguish the contexts. Full marks require the caveat.

EX036 — Model paragraph. "The across-context difference (1.07, z ≈ 2.55) is the more precise estimate, but NMS and KFL differ in telemetry density, staffing model, tooling, incident mix and scenarios. Any of these, alone or together, could explain the smaller KFL effect, so precision does not buy diagnosticity. The within-KFL comparison of high- and low-gap scenarios holds participants, organization, tooling and raters constant and varies only the scenario's telemetry gap, which is the feature the mechanism names. It is less precise (z ≈ 1.70) but it isolates the candidate cause. To strengthen it: preregister the interaction as primary; increase scenarios per gap level; use matched scenario pairs identical except for gap level; and replicate the within-context manipulation at NMS, so that the interaction is tested in both contexts."

EX037 — Model claim. "In two synthetic organizations, a structured handoff artifact reduced receiving responders' comprehension and projection errors where telemetry was sparse (about 0.47 per scenario; practical importance uncertain). Where telemetry was dense, the overall effect was small and uncertain and appeared concentrated in scenarios with high telemetry gaps. Telemetry completeness is a plausible boundary condition; establishing it requires more contexts. Perception errors were not affected in either context." (≈ 85 words.)

EX038 — Model memo (outline). Third context: a synthetic university SOC with student analysts, sparse telemetry, frequent shift handoffs, no structured case tool and low compliance pressure. Mapping: telemetry sparse (→ larger effect), experience low (→ larger), tooling unstructured (→ larger), handoffs frequent (→ larger), compliance low (→ no box-ticking penalty). Prediction: effect at least as large as at NMS, with wide uncertainty because student analysts differ from the professional populations tested (an external-validity gap in population, not only context). Minimal check: a small within-subject replication of the NMS protocol with 20–30 analysts, primary outcome the same, registered prediction "effect ≥ NMS lower bound." The memo must not recommend adoption on the basis of prediction alone.