◇ OpenCyber Academy
Doctoral research · Chapter 5 · English

Chapter 5 — Main study of incident decision quality

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 5 — Main study of incident decision quality

Lesson map

Lesson 05 published. Map, example, EX027.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["5.1 Decision quality as the outcome"]
  N2["5.2 Assignment"]
  N3["5.3 Blinded review"]
  N4["5.4 Preserving raw decisions"]
  N5["5.5 Analysis with uncertainty"]
  N6["5.6 Deviations, missing data and exclusions"]
  N7["5.7 Error analysis and harms"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX027"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["5.1 Decision quality as the outcome"]
  N2["5.2 Assignment"]
  N3["5.3 Blinded review"]
  N4["5.4 Preserving raw decisions"]
  N5["5.5 Analysis with uncertainty"]
  N6["5.6 Deviations, missing data and exclusions"]
  N7["5.7 Error analysis and harms"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX027"]

[Learning outcomes and connection]
    |
    v
[5.1 Decision quality as the outcome]
    |
    v
[5.2 Assignment]
    |
    v
[5.3 Blinded review]
    |
    v
[5.4 Preserving raw decisions]
    |
    v
[5.5 Analysis with uncertainty]
    |
    v
[5.6 Deviations, missing data and exclusions]
    |
    v
[5.7 Error analysis and harms]
    |
    v
[Worked example] --> [Exercise EX027]

Learning outcomes and connection

By the end of this chapter the learner will be able to:

  1. Execute an ethically reviewed study according to its protocol, with randomization, counterbalancing and allocation concealment where appropriate.
  2. Organize blinded review of decisions so that raters judge decision quality without knowing condition or outcome.
  3. Preserve raw decisions and the full chain from raw data to reported numbers.
  4. Analyze effects with uncertainty intervals, relate them to the smallest effect of interest and avoid significance-only reporting.
  5. Conduct error analysis and harms analysis as first-class results, not afterthoughts.
  6. Handle deviations, missing data and multiplicity transparently.

Chapter 4's pilot determined whether the Handoff Study could proceed and fixed the sample size. This chapter runs the main study and analyzes it. Chapter 6 then asks whether the result holds in a different context.

5.1 Decision quality as the outcome

Chapter 1 separated decision quality from outcome (§1.2.4). In a simulation, the separation can be enforced by design: each scenario has defined decision points, and at each point the scenario designer and an expert panel record the information available and the set of defensible options given that information. A decision is judged against that set, not against how the scenario "ends." This approach has two benefits: decisions remain judgeable even when the scenario's later events are random or adversarial, and raters can be blinded to outcomes because there are none to see at the moment of judgment.

Decision quality can have several dimensions. The Handoff Study's primary outcome is the count of comprehension and projection errors (Chapter 3 §3.3). Secondary outcomes include perception errors (for the mechanism pattern, H2), time to first containment decision (H3), and harmful actions — decisions that would cause avoidable harm, such as isolating a host that serves a critical public function when a narrower option was defensible. Harm is pre-specified as a safety outcome, because an intervention that reduces errors of understanding while increasing harmful actions is not an improvement.

5.2 Assignment

Within-subject designs. Each participant experiences all conditions on different scenarios. This controls for individual differences in skill, which are large among responders, and needs fewer participants. It introduces order and carryover effects: participants learn, tire, or carry the SHA's structure into later Standard trials. Mitigations: counterbalance the order of conditions and the pairing of scenarios to conditions (for example, with a Latin square), and present SHA conditions last for half of participants and first for the other half; test for period and carryover effects in analysis.

Between-subject designs. Each participant experiences one condition. There is no carryover, but more participants are needed and randomization must balance skill. Stratify randomization by experience level when samples are small.

Allocation concealment. The person enrolling participants should not know or influence the next assignment. Generate the assignment schedule in advance with a recorded seed, store it where the session facilitator cannot modify it, and reveal each assignment only at session start.

Fidelity. Record whether each session followed the protocol: correct materials, correct timing, no facilitator hints. Fidelity failures are deviations (§5.6), not data to be quietly discarded.

5.3 Blinded review

Raters are the measurement instrument. Blinding keeps them from measuring their expectations.

5.4 Preserving raw decisions

Every number in the results should be traceable to raw records: session recordings or logs (where consented), participant decision forms, raw rater sheets before adjudication, adjudication notes and the analysis scripts. Keep raw records immutable and hashed; derive analysis files by script. This chain is what makes the study auditable, and it is the core of the reproducibility archive in Chapter 8. Raw records also enable later critics (Chapter 7) to recode decisions under alternative outcome definitions.

In the SHA study the raw rater sheets matter most, because error-type coding is where adjudication can quietly change an outcome.

5.5 Analysis with uncertainty

5.5.1 Estimates, not verdicts

Report the estimated effect with a confidence interval and relate the interval to the smallest effect of interest (SEOI). A p-value can be reported, but a conclusion such as "significant" or "not significant" alone discards the information readers need. Four interpretive cases arise:

Interval relative to zero and SEOI Interpretation
Entirely above SEOI Effect present and practically important.
Above zero, includes SEOI Effect present; practical importance uncertain.
Includes zero, excludes SEOI No practically important effect (precise null).
Includes zero and SEOI Inconclusive; study too imprecise.

In plain terms: Where the confidence interval sits relative to zero and to the smallest effect that matters tells you which of these sentences you are allowed to write; "significant" alone is never enough.

5.5.2 Models

Within-subject data with several scenarios per participant are naturally analyzed with mixed-effects models (random effects for participant and scenario; fixed effects for condition, period and possibly experience). The lab below uses simpler paired analyses so that every step can be checked by hand: a paired t-interval and an exact sign-flip permutation test, which assumes only that, under the null hypothesis of no condition effect, the sign of each participant's difference is equally likely to be positive or negative.

5.5.3 Multiplicity

The protocol names one primary outcome. Secondary and exploratory outcomes are reported with their intervals and labeled; if many are tested, the protocol states whether and how it adjusts for multiple comparisons. Undisclosed multiplicity is one of the "forking paths" of Chapter 1.

5.6 Deviations, missing data and exclusions

5.7 Error analysis and harms

An average effect hides where the intervention helps, does nothing or hurts. Error analysis reads the decisions themselves:

Harms analysis treats safety outcomes with the same rigor as benefits. Because harmful actions are rare, a study is usually underpowered to detect an increase. The honest report gives the interval and states that harms of a given size cannot be excluded.

5.8 Worked example: simulated incident cases scored by independent responders under two workflows

All participants, scenarios and results are synthetic. The figure below is a 12-participant teaching excerpt; Chapter 4's planning required 32–71 participants, so the excerpt is deliberately small enough to check by hand. Each participant handled three scenarios under Standard handoff and three under SHA (scenarios seeded from the Harbor Library, Cedar Training Network and Lumen Library Reservations cases, counterbalanced). Two blinded raters coded errors; adjudicated counts are summed over the three scenarios per condition.

"""Chapter 5 offline lab: synthetic within-subject results for the Handoff Study."""
from itertools import product
from statistics import mean, stdev

# participant: (standard_cp_errors, sha_cp_errors, standard_perception, sha_perception, standard_harm, sha_harm)
# cp = comprehension + projection errors summed over 3 scenarios per condition
DATA = {
    "P01": (5, 3, 2, 2, 0, 0), "P02": (4, 4, 1, 1, 0, 1), "P03": (6, 3, 2, 1, 1, 0),
    "P04": (3, 2, 1, 1, 0, 0), "P05": (7, 4, 3, 3, 1, 0), "P06": (4, 3, 1, 2, 0, 0),
    "P07": (5, 5, 2, 2, 0, 0), "P08": (6, 4, 2, 2, 1, 1), "P09": (2, 2, 0, 1, 0, 0),
    "P10": (5, 2, 1, 1, 0, 0), "P11": (4, 3, 2, 2, 0, 1), "P12": (6, 5, 2, 1, 0, 1),
}
T_CRIT_DF11 = 2.201  # two-sided 95% critical value of Student's t with 11 df (standard tables)

def paired(idx_a, idx_b):
    d = [v[idx_a] - v[idx_b] for v in DATA.values()]  # positive = fewer errors with SHA
    m, s = mean(d), stdev(d)
    half = T_CRIT_DF11 * s / len(d) ** 0.5
    return d, m, s, (m - half, m + half)

def sign_flip_p(d):
    obs = abs(sum(d))
    hits = sum(1 for signs in product((1, -1), repeat=len(d))
               if abs(sum(x * s for x, s in zip(d, signs))) >= obs)
    return hits / 2 ** len(d)

if __name__ == "__main__":
    for label, a, b in (("comprehension+projection", 0, 1), ("perception", 2, 3), ("harmful actions", 4, 5)):
        d, m, s, ci = paired(a, b)
        print(f"{label:25s} mean_diff={m:.3f} sd={s:.3f} 95%CI=({ci[0]:.3f}, {ci[1]:.3f}) exact_p={sign_flip_p(d):.4f}")

Expected output (verified with Python 3):

comprehension+projection  mean_diff=1.417 sd=1.165 95%CI=(0.677, 2.157) exact_p=0.0039
perception                mean_diff=0.000 sd=0.603 95%CI=(-0.383, 0.383) exact_p=1.0000
harmful actions           mean_diff=-0.083 sd=0.669 95%CI=(-0.508, 0.341) exact_p=1.0000

Interpretation.

5.9 Practical activity: results section skeleton

Write a results section in this order: participant flow (recruited, consented, randomized, completed, excluded with reasons); fidelity and deviations; rater agreement and blinding check; primary outcome estimate and interval relative to SEOI; mechanism outcome; safety outcome; secondary outcomes; sensitivity analyses; error analysis. Each number must reference the script and data hash that produced it.

In the SHA study, the primary estimate is the difference in comprehension and projection errors per scenario, read against the SEOI of 0.5. If the interval excludes the SEOI, failure condition F3 applies and the section reports a precise null. Placing participant flow first lets a reader judge whether exclusions could explain the estimate before seeing it.

5.10 Reporting the study so that others can judge it

Clinical trials have the most developed reporting guidance for randomized studies. The CONSORT 2010 statement (Schulz, Altman and Moher, 2010) provides a checklist and a participant flow diagram for reporting parallel-group randomized trials. The Handoff Study is a within-subject simulation experiment, not a clinical trial, and CONSORT should not be applied mechanically. But its core demands transfer well and are frequently unmet in security user studies:

A thesis chapter can include a short "reporting checklist" table mapping each of these items to a page. Examiners use such tables; reviewers of later papers will expect them.

5.11 Missing data: mechanisms before methods

Rubin (1976) established the framework still used to reason about missing data, in which the key question is how the probability that a value is missing relates to the data. The modern vocabulary distinguishes three cases:

In incident simulations, MNAR is a realistic threat, because difficulty, confusion and dropout are linked. The protocol should therefore (1) design to prevent missingness (short sessions, breaks, technical rehearsal), (2) record reasons for every missing value, (3) pre-specify the primary handling method, and (4) pre-specify a sensitivity analysis — for example, assigning missing scenario scores the worst observed value in that condition, then the best, and reporting both. If conclusions survive both extremes, missingness is not driving them.

5.12 Multiplicity done properly

Section 5.5.3 noted that secondary outcomes multiply the chance of a false positive. A common correction controls the familywise error rate (for example, Bonferroni), which guards against even one false positive but becomes very conservative as outcomes multiply. Benjamini and Hochberg (1995) introduced control of the false discovery rate — the expected proportion of false positives among the outcomes declared significant — and a simple step-up procedure to achieve it. For a family of secondary outcomes in an exploratory-but-disciplined study, false-discovery-rate control is often the better trade-off.

The procedure: sort the m p-values from smallest to largest; find the largest rank k such that the k-th smallest p-value is at most (k/m)·q, where q is the chosen false discovery rate; declare the outcomes with ranks 1 through k significant. The "step-up" feature means an outcome can be declared significant even if its own p-value exceeds its own rank's threshold, provided a higher-ranked p-value passes.

The Handoff Study also needs a different kind of claim for its safety outcome. "No significant increase in harmful actions" is not evidence of safety; it may reflect imprecision. The correct question is whether any increase is reliably smaller than a pre-declared margin — an equivalence question (§3.13). Using a 90% confidence interval, the two one-sided tests procedure at α = 0.05 concludes equivalence when the whole interval lies inside the margin.

The following lab applies both procedures to synthetic secondary outcomes.

# Synthetic secondary outcomes of the Handoff Study (illustrative only).
# Each: (name, estimate, standard_error) for the SHA-minus-Standard difference.
from statistics import NormalDist
N = NormalDist()
outcomes = [
    ("perception_errors",        -0.10, 0.20),
    ("time_to_containment_min",  -1.80, 0.75),
    ("harmful_actions",           0.05, 0.09),
    ("escalation_errors",        -0.40, 0.17),
    ("documentation_gaps",       -0.55, 0.30),
]
def p_two_sided(est, se):
    return 2 * (1 - N.cdf(abs(est / se)))
ps = [(name, p_two_sided(e, s)) for name, e, s in outcomes]
# Benjamini-Hochberg step-up at q = 0.05
q = 0.05
m = len(ps)
ranked = sorted(ps, key=lambda x: x[1])
k_max = 0
for k, (_, p) in enumerate(ranked, start=1):
    if p <= k / m * q:
        k_max = k
rejected = {name for name, _ in ranked[:k_max]}
print("outcome                   p_raw   BH_reject")
for name, p in ps:
    print(f"{name:<24} {p:6.4f}   {name in rejected}")
# Equivalence (TOST) for harmful actions with margin +/- 0.25 actions per participant
name, est, se = outcomes[2]
margin = 0.25
z = N.inv_cdf(0.95)
lo, hi = est - z * se, est + z * se
print(f"harmful_actions 90% CI: [{lo:.3f}, {hi:.3f}] margin +/-{margin}")
print("equivalent within margin:", lo > -margin and hi < margin)

Expected output:

outcome                   p_raw   BH_reject
perception_errors        0.6171   False
time_to_containment_min  0.0164   True
harmful_actions          0.5785   False
escalation_errors        0.0186   True
documentation_gaps       0.0668   False
harmful_actions 90% CI: [-0.098, 0.198] margin +/-0.25
equivalent within margin: True

Reading the output. Time to containment and escalation errors are declared discoveries at q = 0.05. Note escalation errors: its p-value (0.0186) is the second smallest and passes its rank threshold of 2/5 × 0.05 = 0.02, and because the procedure steps up, the smallest p-value (0.0164) is also declared even though it exceeds its own threshold of 0.01. Documentation gaps (p ≈ 0.067) are not declared, and the results paragraph should describe that estimate with its interval rather than as "no effect." For harmful actions, the 90% interval [−0.098, 0.198] lies within ±0.25, so the study can claim that any increase in harmful actions is smaller than the pre-declared margin — a far stronger safety statement than "not significant." The claim is only as good as the margin: had the protocol set ±0.15, the upper bound 0.198 would exceed it and equivalence could not be claimed.

In plain terms: The first table tells you which secondary results survive a correction for testing five things at once; the last two lines show that the study can positively say the SHA does not meaningfully increase harmful actions, because the plausible range of the effect sits inside the safety margin chosen in advance.

5.13 Analytic flexibility: many analysts and the multiverse

Even with a pre-registered plan, many analysis choices remain: how to treat scenarios with a facilitator deviation, whether to model scenario as a random effect, whether to count a decision with two errors as one or two. Silberzahn et al. (2018) gave the same dataset and research question to many independent analysis teams and found substantial variation in their estimates, arising from reasonable but different analytic choices. Steegen et al. (2016) proposed multiverse analysis: performing the analysis across the full set of reasonable processing and analysis choices and reporting the distribution of results, to make transparent how much conclusions depend on those choices.

For a thesis, a compact multiverse is achievable. List the defensible choices (for example: three rules for handling deviations × two outcome codings × with or without scenario random effects = 12 specifications), run the analysis for each, and report the range of estimates along with the pre-registered one. If all twelve point in the same direction and most exclude zero, the conclusion is robust to analytic choice. If they scatter, the honest conclusion is weaker, and the thesis should say which choice drives the difference. The pre-registered analysis remains primary; the multiverse is a robustness check, not a menu.

5.14 Models for within-subject incident data

The worked example uses paired differences because they are transparent. A full analysis usually fits a mixed-effects model with fixed effects for condition, period and sequence, and random intercepts for participants and scenarios. The model's advantages are practical:

Its risks: models with many random effects may fail to converge on small samples, and analysts then simplify the model in data-dependent ways — another source of flexibility. Pre-specify the model and a fallback sequence ("if the full model does not converge, drop the random slope first, then…"). Count outcomes (errors per scenario) may call for a Poisson or negative-binomial link rather than a normal model; check over-dispersion and pre-specify the choice.

5.15 Qualitative evidence inside the experiment

Numbers show whether the SHA changed errors; they do not show how. Brief, structured debriefs after each condition — "What did you understand to be the main hypothesis when you took over? What did you think had been ruled out? What did you expect to happen next?" — provide process evidence. Two rules keep them from contaminating the primary outcome: hold debriefs after all scored decisions in the scenario are recorded, and analyze transcripts with a codebook developed before unblinding, with codes for the mechanism categories (hypotheses, ruled-out items, gaps, expectations, roles). Agreement between the qualitative pattern and the quantitative mechanism test is convergent evidence; disagreement is a finding to report, not to hide. The sensemaking account (§2.9.4) predicts that debriefs will reveal role confusion that the error scores miss — a specific, checkable expectation.

5.16 Harms analysis in depth

Harm in the Handoff Study is a decision that would cause avoidable damage relative to a defensible alternative. Treating harm as a first-class result means:

  1. Defining harm categories in advance: over-containment (isolating a critical service unnecessarily), under-containment (leaving an attacker foothold that a defensible option would have removed), evidence destruction (actions that would compromise forensic data), and unauthorized disclosure (sharing information beyond its TLP or legal constraints).
  2. Reporting each category by condition, not only the total, because an intervention could reduce under-containment while increasing over-containment.
  3. Reviewing every harmful decision qualitatively. Harmful decisions are rare and individually informative. A short narrative for each, written by a rater blind to condition, often reveals scenario design flaws or plausible failure modes of the aid.
  4. Using equivalence logic for the safety claim, as in §5.12.

Common errors and limits

Exercises — chapter 5

EX027 — Assignment schedule. Objective: design concealed, counterbalanced assignment. Inputs: 24 participants, conditions Standard and SHA, scenarios A–F (three per condition), two experience strata. Constraints: balance order and scenario-condition pairing within strata. Steps: produce the schedule logic (e.g., Latin-square-based); describe concealment; describe fidelity checks. Expected evidence: schedule description and procedures. Rubric: 3 = balanced order and pairing within strata, seed recorded, concealment credible; 2 = one balance missing; 1 = simple alternation; 0 = none.

EX028 — Blinding protocol. Objective: protect rating from expectation. Inputs: sample decision text: "Per the T-gaps field, DNS logs were unavailable for 02:00–04:00, so I did not treat the lack of lookups as evidence of no C2." Constraints: preserve meaning; remove condition cues. Steps: redact the text; list three other cue types; design a blinding check. Expected evidence: redacted text and protocol. Rubric: 3 = cues removed without changing reasoning, check designed; 2 = one cue type missed; 1 = meaning altered; 0 = none.

EX029 — Interpret against SEOI. Objective: draw calibrated conclusions. Inputs: the lab output. Constraints: convert to per-scenario units. Steps: state primary, mechanism and harms conclusions in one sentence each; classify the primary result by the table in §5.5.1. Expected evidence: three sentences and classification. Rubric: 3 = "present, practical importance uncertain," mechanism "consistent with," harms "not excluded"; 2 = one overstatement; 1 = significance-only; 0 = none.

EX030 — Sensitivity analysis. Objective: handle a discovered exclusion. Inputs: after unblinding, the deviation log shows P10 had seen two scenarios in a prior training course. Constraints: the primary analysis stays as planned. Steps: compute the paired analysis without P10 (t critical value 2.228 for 10 df); compute the exact sign-flip p; report both analyses. Expected evidence: numbers and paragraph. Rubric: 3 = mean 1.273, interval (0.531, 2.014), p 0.0078 and correct reporting of both; 2 = numbers right, reporting wrong; 1 = replaces primary; 0 = none.

EX031 — Error analysis plan. Objective: read decisions, not averages. Inputs: the Handoff Study. Constraints: at least five questions; include possible new errors caused by SHA. Steps: write questions, data needed and how each answer would change the claim. Expected evidence: plan. Rubric: 3 = includes anchoring/overtrust, per-scenario and per-participant analyses, links to claim; 2 = generic; 1 = averages only; 0 = none.

EX032 — Results paragraph. Objective: write publishable, calibrated results. Inputs: lab output, your EX030 result. Constraints: 200–300 words; every number traceable. Steps: follow §5.9 order for primary, mechanism, harms and sensitivity. Expected evidence: paragraph. Rubric: 3 = calibrated, complete, harms reported, sensitivity included; 2 = one omission; 1 = overclaims; 0 = none.

Chapter synthesis

A main study earns trust through execution discipline: concealed and counterbalanced assignment, blinded raters with measured agreement, preserved raw decisions, estimates with intervals interpreted against a predeclared smallest effect of interest, and equal rigor for harms. In the synthetic excerpt, SHA reduced comprehension and projection errors with uncertain practical importance, left perception errors unchanged as predicted, and could not exclude a small increase in harmful actions. Chapter 6 asks whether this pattern survives a different organization with different staffing and telemetry.

Continue studying — chapter 05 of 08

← Chapter 04 · Course chapter index · Chapter 06 →

Continue to the next authored lesson.

Worked answers

EX027 — Model assignment schedule. Split 24 participants into two experience strata of 12. Within each stratum, use a balanced design crossing order (SHA-first vs. Standard-first) with scenario-set allocation (set 1 = A, B, C; set 2 = D, E, F under SHA vs. under Standard), giving four sequences; assign three participants per sequence per stratum. Within each block, order the three scenarios by a 3×3 Latin square so each scenario appears in each position equally often. Generate the sequence list with a recorded seed before recruitment begins, held by a team member not involved in sessions; the facilitator opens each participant's assignment only at session start. Fidelity: checklist per session (correct packets, timer, no hints), with a second person auditing 20% of recordings where consent permits. Full credit requires balance within strata and credible concealment.

EX028 — Model blinding. Redacted: "DNS logs were unavailable for 02:00–04:00, so I did not treat the lack of lookups as evidence of no C2." (The information is kept; the cue "Per the T-gaps field" is removed. Because Standard participants could also learn of the gap from free text, the reasoning itself does not reveal condition.) Other cues: SHA field names (H-ruled-out, Expect), characteristic formatting (bulleted fields), and references to "the template" or "the form"; also differences in length if SHA prompts longer answers — mitigate by transcribing to a neutral format. Blinding check: after rating, raters guess condition for a random 20% of items; report accuracy with an interval against 50%.

EX029 — Model interpretation. Primary: "SHA reduced comprehension and projection errors by about 0.47 per scenario (95% CI about 0.23 to 0.72); an effect is likely present, but whether it reaches the predeclared practical threshold of 0.5 is uncertain" — the "above zero, includes SEOI" case. Mechanism: "Perception errors did not differ (0.00 per three scenarios, CI −0.38 to 0.38), consistent with the predicted pattern; modest effects cannot be excluded." Harms: "Harmful actions were slightly more frequent with SHA (−0.08 per three scenarios, CI −0.51 to 0.34); an increase cannot be excluded."

EX030 — Model sensitivity. Without P10 the differences are [2, 0, 3, 1, 3, 1, 0, 2, 0, 1, 1]: mean 1.273, SD 1.104, 95% CI (0.531, 2.014) over three scenarios (≈0.18 to 0.67 per scenario), exact sign-flip p 0.0078. Report: "In the preregistered primary analysis including all participants, the reduction was 1.42 (0.68 to 2.16). A deviation log entry recorded after unblinding showed that P10 had prior exposure to two scenarios; excluding P10 in a sensitivity analysis gives 1.27 (0.53 to 2.01). Conclusions are unchanged in direction; the estimated effect is somewhat smaller and practical importance remains uncertain." Zero credit for silently replacing the primary analysis.

EX031 — Model error analysis plan. (1) Per-scenario error distribution by condition — does the effect concentrate in scenarios with large telemetry gaps (supporting the mechanism) or appear uniformly (suggesting a general effect)? (2) Per decision point — which points drive the difference? (3) New error types under SHA — anchoring on H-current, or overtrust in an incomplete T-gaps list (code a specific "inherited error" category). (4) Harmful actions — read all cases; were they preceded by SHA content? (5) Participant heterogeneity — do novices and veterans differ (links to the skilled-intuition account)? (6) Rater disagreement items — are disagreements concentrated in one error type, indicating construct ambiguity? Each answer either supports, narrows or challenges H1/H2.

EX032 — Model results paragraph. "Twelve participants completed all six scenarios (teaching excerpt). Raters' pre-adjudication agreement is reported in Table X. In the preregistered primary analysis, participants made 1.42 fewer comprehension and projection errors over three scenarios with SHA than with Standard handoffs (95% CI 0.68 to 2.16; exact sign-flip p = 0.004), equivalent to 0.47 per scenario (0.23 to 0.72). The interval includes the predeclared smallest effect of interest (0.5 per scenario), so practical importance is uncertain. Perception errors did not differ (0.00; −0.38 to 0.38), consistent with the predicted mechanism. Harmful actions were slightly more frequent with SHA (−0.08; −0.51 to 0.34); an increase cannot be excluded, and all seven harmful actions (four with SHA, three with Standard) are examined in the error analysis. A sensitivity analysis excluding one participant with prior scenario exposure gave 1.27 (0.53 to 2.01). Analyses: ch05_lab.py, data hash recorded in the run log."