Chapter 3 — Original response-science contribution proposal

Chapter 3 — Original response-science contribution proposal

Lesson map

Lesson 03 published. Map, example, EX015.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["3.1 What counts as a contribution"]
  N2["3.2 Mechanism before measurement"]
  N3["3.3 Hypotheses, outcomes and the smallest effect of"]
  N4["3.4 Baselines and comparisons"]
  N5["3.5 Alternative explanations and how to separate the"]
  N6["3.6 Predeclared failure conditions"]
  N7["3.7 Risk register"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX015"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["3.1 What counts as a contribution"]
  N2["3.2 Mechanism before measurement"]
  N3["3.3 Hypotheses, outcomes and the smallest effect of"]
  N4["3.4 Baselines and comparisons"]
  N5["3.5 Alternative explanations and how to separate the"]
  N6["3.6 Predeclared failure conditions"]
  N7["3.7 Risk register"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX015"]

[Learning outcomes and connection]
    |
    v
[3.1 What counts as a contribution]
    |
    v
[3.2 Mechanism before measurement]
    |
    v
[3.3 Hypotheses, outcomes and the smallest effect of]
    |
    v
[3.4 Baselines and comparisons]
    |
    v
[3.5 Alternative explanations and how to separate the]
    |
    v
[3.6 Predeclared failure conditions]
    |
    v
[3.7 Risk register]
    |
    v
[Worked example] --> [Exercise EX015]

Learning outcomes and connection

By the end of this chapter the learner will be able to:

1. Classify a proposed contribution as empirical, artifact, method or theory, and state what kind of evidence each requires.
2. Specify a causal mechanism precisely enough that it predicts where and how an effect should appear.
3. Write hypotheses with operational outcomes, a smallest effect of interest and directional predictions.
4. Choose a credible baseline and an active comparison that rule out trivial explanations.
5. List alternative explanations and design features that separate them from the proposed mechanism.
6. Predeclare failure conditions and a risk register for the project.

Chapter 2 produced a falsifiable gap: whether a structured handoff artifact changes the correctness of a receiving responder's decisions under incomplete telemetry. This chapter turns that gap into a proposal that a committee could approve and a hostile reviewer could not easily dismiss. Chapter 4 will test whether the baseline and the procedure are viable.

3.1 What counts as a contribution

A doctoral contribution changes what a competent community has warrant to believe or do. In incident-response research it usually takes one of four forms, often combined:

In plain terms: A contribution's type determines the evidence it must produce; an artifact needs evidence that it changes behavior, not just that people like it.

A common weakness is to propose an artifact and evaluate only whether people like it. Preference is a legitimate outcome for usability, but it is not evidence that the artifact improves decisions. Another weakness is to call a method "validated" because it was used once. State which type the thesis claims, and match evidence to type.

3.2 Mechanism before measurement

A mechanism explains how an intervention would produce an effect. Specifying it does three jobs: it tells the researcher which outcomes to measure, it predicts where the effect should be large or absent, and it gives critics something precise to attack.

The Handoff Study's mechanism, grounded in the situation-awareness account of Endsley (1995), can be written as a chain:

1. During an incident, the sending responder holds working hypotheses, knowledge of what has been ruled out, knowledge of which telemetry is missing, and expectations about what will happen next.
2. Free-text handoffs tend to transmit current facts and pending tasks but omit ruled-out hypotheses, known telemetry gaps and expectations (this premise is supported only at qualitative level by synthetic study S3 in Chapter 2 and must be checked in the pilot).
3. Without these, the receiving responder (a) re-investigates eliminated hypotheses, (b) reads absence of evidence in missing telemetry as evidence of absence (Chapter 1 §1.2.3), and (c) fails to anticipate scheduled or likely developments.
4. A Structured Handoff Artifact (SHA) with required fields for these items transmits them.
5. Therefore receiving responders with an SHA will make fewer errors of comprehension (misreading the significance of missing data) and projection (failing to anticipate), but not necessarily fewer errors of perception (failing to see a signal present in the data).

The final clause is what makes the mechanism scientifically useful: it predicts a pattern, not just an improvement. If SHA reduced perception errors as much as comprehension errors, the proposed mechanism would be in doubt, and a more general explanation — for example, that any additional structure focuses attention — would gain support.

3.2.1 The SHA specification (synthetic artifact)

In plain terms: Every field in the handoff artifact exists because the mechanism says a particular kind of error is caused by missing information; a field without a mechanism link is decoration.

3.3 Hypotheses, outcomes and the smallest effect of interest

A hypothesis should name the population, condition, comparison, outcome and direction. For the Handoff Study:

• H1 (primary). Among experienced responders receiving a simulated incident at a handoff point under incomplete telemetry, those receiving an SHA will make fewer comprehension- and projection-type decision errors per scenario than those receiving the organization's standard free-text handoff.
• H2 (mechanism pattern). The reduction in H1 will be larger than any reduction in perception-type errors.
• H3 (secondary). SHA will not increase time to the first containment decision by more than a pre-specified margin (a non-inferiority hypothesis on speed).

Outcome definitions. Errors are coded by two blinded expert raters against a scenario-specific reference that lists, for each decision point, the information available at that moment (Chapter 1 §1.2.4) and the defensible options. Each error is assigned to perception (relevant signal present in the materials but not used), comprehension (signal used but significance misread, including treating missing telemetry as negative evidence) or projection (failure to anticipate a development signaled in the materials). The codebook, examples and rater-training procedure are part of the protocol, and agreement is reported (Chapter 1 §1.6.4).

Smallest effect of interest (SEOI). Before any data, the researcher states the smallest difference that would matter in practice. Here the reasoning might be: "A reduction of fewer than 0.5 comprehension/projection errors per scenario would not justify the time cost of completing an SHA during a busy shift." The SEOI must be justified by practical consequence, not chosen to make a study feasible, and it drives the sample-size reasoning in Chapter 4. A SEOI also lets a null result be informative: if the confidence interval excludes effects as large as the SEOI, the study supports "no practically important effect," not merely "no significant effect."

3.4 Baselines and comparisons

A baseline is what the intervention is compared against. A weak baseline makes any intervention look good. Three principles:

1. Use what practitioners actually do, not a degraded version. The baseline is NMS-CSIRT's standard free-text handoff, written by responders who are given the same time as the SHA group. Comparing SHA with "no handoff" would test a different and less interesting question.
2. Add an active control when a trivial explanation is plausible. SHA asks senders to spend more time and write more. An attention-control condition — a free-text handoff with the same minimum length and time allocation but no required fields — separates "structure" from "more effort."
3. Reproduce published baselines before claiming improvement. If a prior study reports a baseline error rate or a benchmark score, it should be reproduced under your conditions (Chapter 4). The Arp et al. (2022) pitfall of inappropriate baselines applies to human-study designs as well as to machine learning.

The proposal thus has three conditions: Standard (free text, usual practice), Attention-control (free text with equalized effort) and SHA. H1 compares SHA with Standard; a secondary comparison of SHA with Attention-control tests whether structure matters beyond effort.

3.5 Alternative explanations and how to separate them

For every proposed mechanism, list rival explanations and the design feature that addresses each:

In plain terms: For each competing explanation of a positive result, the design includes a specific feature that would expose it; the table is the study's internal-validity defence in compact form.

This table becomes the core of the protocol's internal-validity section and is often the part of a proposal reviewers read most closely.

3.6 Predeclared failure conditions

A proposal should say, before data are collected, what results would count against the contribution and what the researcher would do. For the Handoff Study:

In plain terms: These are the results that would force the claim to shrink, written down before the data exist so that nobody can explain them away afterwards.

Each condition maps to a specific narrowing of the claim. That mapping is what makes the contribution falsifiable in practice rather than in principle.

3.7 Risk register

Doctoral projects fail more often from access, recruitment and measurement problems than from wrong hypotheses. A risk register names each risk, its likelihood, its impact and the mitigation.

In plain terms: The main risks are practical (recruitment, data access, rater time), and each has a named mitigation that can be checked during the pilot.

3.8 Evaluation criteria for the contribution

State how a committee should judge the thesis, regardless of the direction of results:

1. Was the baseline reproduced and the procedure piloted with measured rater agreement (Chapter 4)?
2. Did the main study test H1–H3 as preregistered, with deviations logged (Chapter 5)?
3. Was the effect, or its absence, tested in a second context (Chapter 6)?
4. Did the claims survive or get narrowed by adversarial critique (Chapter 7)?
5. Are the artifacts (SHA, codebook, scenarios, analysis code, synthetic data) available at the tiers promised (Chapter 8)?

A proposal that commits to these criteria makes a negative result publishable and useful.

3.9 Worked example: from gap statement to proposal summary

Synthetic. The candidate writes a one-page proposal summary:

Title. Structured handoffs and decision errors under incomplete telemetry: a preregistered simulation study.

Contribution type. Empirical and artifact, with a theory test (error-type pattern).

Problem. Context is reported to be lost when incidents change hands (qualitative evidence only), and missing telemetry invites absence-of-evidence errors (§1.2.3). No study in our documented search compares handoff formats on decision correctness.

Mechanism. SHA transmits ruled-out hypotheses, telemetry gaps and expectations, which should reduce comprehension and projection errors but not perception errors.

Design. Within-subject simulation; three conditions (Standard, Attention-control, SHA); six synthetic scenarios seeded from the Harbor Library, Cedar Training Network and Lumen Library Reservations cases; standardized handoffs written by trained senders; counterbalanced order; blinded raters.

Hypotheses. H1, H2, H3 as in §3.3. SEOI 0.5 errors per scenario. Non-inferiority margin for time stated in protocol.

Failure conditions. F1–F6.

Ethics. Participants are consenting professionals working only on synthetic material; no incident records are used; consent addresses voluntary participation and non-disclosure of individual performance to employers.

Artifacts. SHA template, scenarios, codebook, analysis code and anonymized decision data released under tiers T0–T2 and, if consent permits, de-identified T3.

The summary fits on one page and every element maps to a section of the full protocol.

3.10 Practical activity: mechanism stress test

Take your own proposed mechanism and attempt to break it in writing, in three passes:

1. Premise pass. For each link in the chain, state the evidence for it and its ladder level. Mark any link supported only by intuition.
2. Pattern pass. Write down at least one outcome the mechanism predicts should not change. If you cannot, the mechanism is not yet specific enough.
3. Rival pass. For each rival explanation, write what result would favor it over your mechanism, and confirm your design can observe that result.

Record the stress test in the protocol appendix; reviewers value seeing that the researcher attacked their own idea first.

3.11 Drawing the causal model before choosing measures

The mechanism chain in §3.2 is a narrative. Before data collection, it is worth translating it into a simple causal diagram: nodes for variables, arrows for assumed direct causal influence. No special software or notation is needed; the value lies in forcing every assumption into the open. For the Handoff Study, a minimal diagram contains:

• Handoff format (Standard or SHA) → content transmitted (hypotheses, ruled-out items, gaps, expectations) → receiver comprehension and projection → decision quality.
• Receiver experience → comprehension and projection, and → decision quality directly (experts may compensate for poor handoffs).
• Telemetry completeness of the scenario → comprehension and projection (gaps make understanding harder) and → the value of transmitted gap information (an effect modifier).
• Sender effort and time available → content transmitted (a rushed sender transmits less in either format).
• Scenario difficulty → decision quality.

Three design consequences follow from drawing it.

1. Which variables must be controlled by design. Scenario difficulty affects the outcome but not the format; balancing scenarios across conditions (§5.2) removes it as a confound. Sender effort affects content in both arms; if the SHA takes longer to write, a study that gives senders unlimited time measures the SHA under unrealistic conditions. The protocol must fix the sender's time budget equal across arms.
2. Which variables are mediators and must not be "adjusted away." Content transmitted lies on the causal path. Adjusting the primary analysis for the number of fields completed would remove part of the effect the study is trying to estimate. Mediators belong in a separate, labeled mechanism analysis.
3. Which variables are moderators to pre-specify. Experience and telemetry completeness are candidates. Pre-specifying them, with predicted directions, turns post-hoc subgroup hunting into a test.

The diagram also exposes what the study cannot learn. Nothing in it represents the organizational pressures discussed in §2.9.5. That omission is acceptable for a simulation study but must appear in the limitations as a boundary on the claim.

3.12 Construct validity of "decision quality"

The primary outcome is only as credible as the argument that it measures what the thesis says it measures. A construct-validity argument for rater-scored decision errors has several strands, each of which can be planned in the proposal.

• Content. The defensible-option sets for each decision point come from an expert panel. Document who the experts are (role, years of practice, sector), how disagreements were resolved, and how many decision points were dropped because experts could not agree. Dropped points are informative: they mark where "correct" is genuinely contested.
• Response process. Do raters score what the rubric intends? Think-aloud sessions with two raters scoring a sample of decisions can reveal rubric ambiguities before the main study.
• Internal structure. If comprehension and projection errors are distinct constructs, they should not be perfectly correlated, and the study's mechanism claim requires that they be scored separately with acceptable agreement for each.
• Relations to other variables. Error counts should be higher for novices than experts and higher in scenarios with more telemetry gaps. If they are not, either the measure or the scenarios are suspect. These are known-groups checks and can be run in the pilot.
• Consequences. If decision-error scores were ever used for performance evaluation of real responders, the incentive to game them would change their meaning. The thesis should state that its scores are research instruments, not personnel measures.

3.13 Justifying the smallest effect of interest

Section 3.3 introduced the smallest effect of interest (SEOI). Doctoral committees increasingly ask why that value. Lakens (2022) discussed several approaches to justifying sample size, among them planning for a smallest effect of interest, accepting a sample size fixed by resource constraints and reporting the resulting precision honestly, and using heuristics; he cautioned against unjustified conventions. Applying that discussion to the Handoff Study suggests four defensible justifications, which can be combined:

1. Practitioner elicitation. Ask several experienced incident leads how large a reduction in misread situations would make them adopt a new handoff format, given its cost in sender time. Record the elicitation protocol and the spread of answers, not only the median.
2. Cost–benefit anchoring. If the SHA adds two minutes per handoff, how many errors must it prevent per handoff to be worth the time? Even rough numbers make the SEOI an argument rather than a guess.
3. Measurement resolution. An effect smaller than the scoring instrument can reliably detect (given rater disagreement) is not a meaningful target.
4. Resource-bounded precision. If only a fixed number of qualified participants can be recruited, state the precision that sample buys and accept that some true effects of practical interest may not be distinguishable from zero.

The SEOI also defines what counts as "no meaningful effect." With an SEOI declared, the study can use equivalence testing: Lakens (2017) presented a practical introduction to the two one-sided tests (TOST) procedure, which asks whether the effect is reliably smaller than the bounds set by the SEOI. Without equivalence logic, a null result can only be called "not significant," which is uninformative. Chapter 5 applies TOST to the safety outcome.

3.14 Designing to discriminate between rival theories

Section 2.3.6 argued that the best studies test points where theories disagree. The proposal should show concretely how the Handoff Study does this. Using the deeper reading of Chapter 2 §2.9:

In plain terms: Each row names a competing explanation, what it alone predicts, and the piece of the design that can tell it apart from the others. A result that fits only the first row supports the mechanism; a result that fits the second row means the template's structure did not matter.

The word-count-matched condition deserves comment because it is the cheapest and most often omitted control. If SHA handoffs are on average 40% longer than free-text handoffs, any benefit could be due to length. Instructing Standard-condition senders to write handoffs of similar length, or comparing within length bands, separates "structure" from "more words." Omitting the control does not invalidate the study, but it limits the claim to "the SHA as typically written," not "the SHA's structure."

3.15 Extensions the proposal should anticipate but not promise

A proposal earns credibility by being narrow. Still, a committee will ask where the work leads. Two extensions follow naturally from Chapter 2 and should appear as clearly labeled future directions, with their own risks.

Cross-organizational handoffs. Coordination research (Comfort, 2007; Bigley and Roberts, 2001) locates many crisis failures at organizational boundaries. Extending the SHA to handoffs between a victim organization and an external responder introduces new variables: information withheld for legal reasons, TLP markings, differing vocabularies, and divergent incentives. The mechanism may be the same; the confounds multiply. Recruitment requires two cooperating organizations per dyad, which makes the sample far harder to obtain.

Model-assisted handoffs. A summarization model could draft the SHA from tickets and chat. The reliance literature (§2.11) predicts both benefits (completeness) and risks (unchecked errors). An extension study would need seeded errors and stratified reporting by model correctness. Its ethics review must address the processing of incident communications by a model, including where the model runs and whether inputs are retained — a data-protection question as much as a research-design one.

Labeling these as future work is not modesty for its own sake: it prevents scope creep from weakening the core study and gives the defense (Chapter 8) a ready answer to "what next?"

3.16 Defending the proposal

The proposal defense or qualifying review tests whether the candidate understands their own design. Prepare short, evidence-based answers to the questions most likely to be asked:

1. Why simulation rather than real incidents? Because real incidents cannot be randomized, their decision quality cannot be judged blind to outcome, and access to raw decision records is rarely granted. Simulation buys internal validity at the price of ecological validity, and Chapter 6 partially repays that price.
2. Why should practitioners care about rater scores? Because the scores are anchored to defensible-option sets elicited from practitioners and validated by known-groups checks (§3.12).
3. What if the effect is null? Then equivalence testing against the SEOI (§3.13) tells whether the SHA is practically useless or the study was too imprecise; both are reportable results with pre-declared consequences (§3.6).
4. What if experts and novices differ? That is a pre-specified moderator with a theory-based prediction (§3.14), not a post-hoc rescue.
5. What is the contribution if the mechanism is wrong but the effect is real? An empirical contribution with a narrowed theoretical claim; the failure conditions in §3.6 say so in advance.

Writing these answers before data collection often reveals design gaps that are still cheap to fix.

Common errors and limits

• Proposing an artifact and measuring only satisfaction.
• A mechanism that predicts improvement everywhere, so no pattern could count against it.
• A "no intervention" baseline when practitioners already do something.
• Omitting an attention-control condition when effort is a plausible explanation.
• Choosing the smallest effect of interest to fit the available sample.
• Failure conditions that lead to no change in claims.
• Limit: the SHA, scenarios and premises in this chapter are synthetic teaching material; their real-world plausibility must be checked by pilot work and practitioner review.

Exercises — chapter 3

EX015 — Contribution typing. Objective: match claims to evidence. Inputs: four synthetic thesis claims: (a) "We release an open-source playbook engine"; (b) "Our scoring rubric for containment decisions is reliable"; (c) "Responders rely on intuition more for familiar incident types"; (d) "Our playbook engine reduces containment errors." Constraints: use §3.1. Steps: type each claim; state the minimum evidence needed; identify one claim that is really two. Expected evidence: table. Rubric: 3 = correct types, appropriate evidence, (a)/(d) relationship noted; 2 = one error; 1 = generic evidence; 0 = none.

EX016 — Mechanism specification. Objective: write a pattern-predicting mechanism. Inputs: an intervention of your choice (e.g., an automated timeline view, a pairing policy for night shifts). Constraints: at least four links; at least one predicted non-effect. Steps: write the chain; attach evidence level to each link; state the non-effect. Expected evidence: mechanism chain and justification. Rubric: 3 = specific chain with testable non-effect and honest evidence levels; 2 = non-effect weak; 1 = no pattern; 0 = none.

EX017 — Hypotheses and SEOI. Objective: operationalize hypotheses. Inputs: your EX016 mechanism. Constraints: one primary, one mechanism-pattern and one safety or non-inferiority hypothesis; a justified SEOI. Steps: write hypotheses with population, condition, comparison, outcome, direction; define outcomes operationally; justify SEOI by consequence. Expected evidence: hypotheses block. Rubric: 3 = complete, operational, SEOI justified by practice; 2 = one element vague; 1 = SEOI arbitrary; 0 = none.

EX018 — Baseline critique. Objective: detect weak baselines. Inputs: a synthetic design: "We compared our AI-generated incident summary with no summary, using 20 student volunteers; the AI group answered more scenario questions correctly." Constraints: propose at least two improved comparison conditions. Steps: identify baseline problems; propose Standard and Active-control conditions; state which rival each addresses. Expected evidence: critique and redesign. Rubric: 3 = identifies no-summary straw man, population mismatch, effort/length confound; proposes human-written summary and length-matched control; 2 = one missed; 1 = superficial; 0 = none.

EX019 — Rival-explanation table. Objective: design against alternatives. Inputs: your EX016–EX017 work. Constraints: at least six rivals. Steps: build a table like §3.5. Expected evidence: table. Rubric: 3 = six or more plausible rivals with concrete design features; 2 = four or five; 1 = generic; 0 = none.

EX020 — Failure conditions and risk register. Objective: make the contribution falsifiable in practice. Inputs: your proposal. Constraints: at least five failure conditions with consequences; at least five risks. Steps: write both tables; check that each failure condition narrows a specific claim. Expected evidence: two tables. Rubric: 3 = each condition maps to a claim change, risks have mitigations; 2 = one or two conditions lack consequence; 1 = vague; 0 = none.

Chapter synthesis

A strong proposal states a mechanism that predicts a pattern, tests it against a realistic baseline and an active control, lists rival explanations with design features that separate them, sets a smallest effect of interest by practical consequence, and predeclares what results would narrow or refute the claim. The Handoff Study now has hypotheses H1–H3, three conditions and failure conditions F1–F6. Chapter 4 asks whether its baselines can be reproduced and its procedure works before the main study is run.

Continue studying — chapter 03 of 08

← Chapter 02 · Course chapter index · Chapter 04 →

Continue to the next authored lesson.