Chapter 2 — Research frontier in incident response systems

Chapter 2 — Research frontier in incident response systems

Lesson map

Lesson 02 published. Map, example, EX012.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["2.1 Why the literature is harder to find in incident"]
  N2["2.2 Systematic scoping"]
  N3["2.3 Competing theoretical accounts of incident decis"]
  N4["2.4 Unresolved evidence"]
  N5["2.5 The contribution test"]
  N6["2.7 Practical activity: build a literature map"]
  N7["2.8 From map to frontier"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX012"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["2.1 Why the literature is harder to find in incident"]
  N2["2.2 Systematic scoping"]
  N3["2.3 Competing theoretical accounts of incident decis"]
  N4["2.4 Unresolved evidence"]
  N5["2.5 The contribution test"]
  N6["2.7 Practical activity: build a literature map"]
  N7["2.8 From map to frontier"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX012"]

[Learning outcomes and connection]
    |
    v
[2.1 Why the literature is harder to find in incident]
    |
    v
[2.2 Systematic scoping]
    |
    v
[2.3 Competing theoretical accounts of incident decis]
    |
    v
[2.4 Unresolved evidence]
    |
    v
[2.5 The contribution test]
    |
    v
[2.7 Practical activity: build a literature map]
    |
    v
[2.8 From map to frontier]
    |
    v
[Worked example] --> [Exercise EX012]

Learning outcomes and connection

By the end of this chapter the learner will be able to:

1. Scope a literature search on an incident-response question systematically, with a recorded protocol that another researcher could repeat.
2. Read primary studies for the claim they can support, using Chapter 1's evidence ladder and failure-mode catalog.
3. Distinguish competing theoretical accounts of incident decision-making and say what each predicts differently.
4. Build a literature map that shows where evidence is strong, weak, conflicting or absent.
5. State a research gap as a falsifiable proposition rather than a topic, and test whether filling it would be a contribution.

Chapter 1 established how to judge a single claim. This chapter applies that judgment to a body of work. Its output — a literature map and a falsifiable gap statement — becomes the input to Chapter 3's contribution proposal.

2.1 Why "the literature" is harder to find in incident response

Incident-response knowledge is spread across very different kinds of publication:

In plain terms: No single source type is trustworthy on its own: peer-reviewed studies are rigorous but narrow, practitioner sources are realistic but unaudited, so a review must state in advance how it will weigh each.

A doctoral literature review must decide, in advance, which of these it will treat as evidence for an empirical claim, which as context (what practitioners believe or recommend), and which as case material. Mixing them without labels is how a review ends up citing a vendor percentage as if it were a population estimate, or a standard's recommendation as if it were a tested finding.

2.2 Systematic scoping

2.2.1 Systematic review, scoping review, narrative review

A systematic review answers a focused question by locating, appraising and synthesizing all eligible studies using a predefined, reproducible method. A scoping review maps the extent and nature of evidence on a broader topic without necessarily appraising quality in depth. A narrative review synthesizes selectively according to the author's judgment. All three are legitimate; the integrity failure is to present a narrative review as if it were systematic.

The PRISMA 2020 statement (Page et al., 2021), which updated the 2009 PRISMA statement, provides a reporting guideline for systematic reviews, including a checklist and a flow diagram that records how many records were identified, screened, excluded (with reasons) and included. PRISMA was developed mainly for health research, and some items (for example, risk-of-bias tools designed for clinical trials) need adaptation for security. Its core discipline transfers directly: say where you searched, with which terms, when, how records were screened and by whom, and why each excluded full text was excluded.

2.2.2 A search protocol for an IR question

Write the search protocol before searching, and version it like the research protocol in Chapter 1.

1. Question in structured form. For decision-support research, a useful frame is population (who decides — tier-1 analysts, incident leads), intervention or exposure (the decision aid, process or condition), comparison (what they would otherwise do), outcome (decision quality, time, error, workload) and context (SOC, CSIRT, MSSP, exercise, field).
2. Sources. Bibliographic databases and digital libraries of the main security, human-factors and computing publishers; proceedings of usable-security venues; standards bodies; selected grey literature with explicit inclusion criteria.
3. Search strings. Synonym sets for each element, e.g. ("incident response" OR "security operations" OR SOC OR CSIRT) AND (triage OR "alert" OR "decision support" OR handoff) AND (study OR evaluation OR experiment OR interview). Record the exact string per database and the date run.
4. Eligibility criteria. Empirical study (any design) of human or human–machine decisions in a security-response setting; excludes purely algorithmic detection papers unless they evaluate effects on analyst decisions.
5. Screening. Two screeners independently screen titles and abstracts and then full texts, with disagreement resolved by discussion and agreement recorded (Chapter 1's kappa applies).
6. Extraction. A form recording design, setting, sample, measures, main claim, the claim's evidence-ladder level and key threats.
7. Snowballing. Backward (references) and forward (citations) searches from included studies, recorded separately.
8. Stopping rule. State when the search ends — for example, when two successive snowballing rounds yield no new eligible study.

2.2.3 Appraisal without a clinical toolkit

Security lacks widely adopted risk-of-bias instruments for human studies. Build an appraisal form from Chapter 1: unit of analysis; sampling frame and selection; construct definitions and measure validity; comparison condition; blinding; rater agreement; base rates; time-respecting splits for learned components; artifact availability; ethics statement. For machine-learning components, the ten pitfalls of Arp et al. (2022) serve as a ready checklist. Record each judgment with a quotation or page reference so that a second appraiser can check it.

2.3 Competing theoretical accounts of incident decision-making

A gap is meaningful only relative to theory: what do we expect to happen, and why? Several traditions offer different accounts of how responders decide, and they make different predictions about what would help.

2.3.1 Situation awareness

Endsley (1995) proposed a theory of situation awareness in dynamic systems with three levels: perception of elements in the environment, comprehension of their meaning and projection of their future status. Applied to response, the theory predicts that decision errors often arise from failures of perception (the relevant signal was not seen), comprehension (seen but misread) or projection (understood but its consequences not anticipated), and that aids should be designed for the level at which failures occur. Its prediction for a handoff: the receiving responder's errors will cluster at comprehension and projection if the handoff transmits facts without their significance.

2.3.2 Naturalistic decision-making and skilled intuition

Kahneman and Klein (2009), writing from the heuristics-and-biases and naturalistic decision-making traditions respectively, argued that skilled intuition can develop when the environment is sufficiently regular to be predictable and when people have adequate opportunity to learn those regularities through practice and feedback. Applied to response, this predicts that intuition will be reliable for frequent, well-fed-back tasks (routine alert dispositions in a stable environment) and unreliable for rare, novel or poorly fed-back ones (a first-of-its-kind intrusion, or dispositions never checked against outcome). It predicts that decision aids will help most where feedback is poor.

2.3.3 Sociotechnical and organizational accounts

Fieldwork such as the six-month anthropological study of a corporate SOC treats response performance as an emergent property of human, technical and managerial factors in cycles — for example, morale, skill growth, automation and management metrics reinforcing one another. The interview study by Kokulu et al. (2019) found disagreements between SOC managers and analysts. These accounts predict that a decision aid will succeed or fail depending on organizational fit — whether it changes workload, autonomy, metrics and trust — not only on its information content.

2.3.4 Detection-centric and automation accounts

Much security research implicitly treats response quality as downstream of detection quality: better alerts, fewer false positives, better prioritization. This account predicts that improving classifier performance improves response. Sommer and Paxson (2010) and Arp et al. (2022) show why lab performance may not translate to operations, and Alahmadi et al. (2022) show that "false positive" in practice often means a benign trigger, which a better classifier of maliciousness would not remove.

2.3.5 Safety-science accounts

The complex-systems view summarized in How Complex Systems Fail treats failure as the joint product of multiple contributing factors and treats practitioners' adaptations as a source of safety as well as risk. It predicts that interventions that constrain responders' adaptive capacity (rigid playbooks, punitive metrics) may reduce some errors while increasing brittleness in novel incidents.

2.3.6 Using competing accounts productively

The accounts are not mutually exclusive, but they make different predictions about the same intervention. That is what makes them scientifically useful. Consider a structured handoff note for incidents transferred between shifts:

In plain terms: The theories disagree about what a handoff note should change; a good study measures the thing on which their predictions differ, not only whether the note "helps."

A study designed to discriminate among these predictions contributes more than one designed only to show that "the note helps."

2.4 Unresolved evidence

An open question exists when evidence is absent, weak, conflicting or untransferred (strong in one setting but untested in the setting of interest). Distinguish these carefully:

• Absent: no eligible study tests the claim. Verify by a documented search, not by a quick query.
• Weak: studies exist but cannot support the claim at the required level (e.g., only self-report, only single site, no comparison).
• Conflicting: well-conducted studies disagree. The interesting question is usually why — different populations, measures or contexts.
• Untransferred: strong evidence from another domain (aviation, medicine) has not been tested in security response, and there are reasons to doubt transfer (adversarial environment, different feedback structure).

The IEEE Access survey of SOC research is an example of a published claim about where evidence is thin: it reports that research tends to address human and technological aspects while neglecting the processes, especially non-technical ones, that connect them. A doctoral researcher should treat such a statement as a starting hypothesis about the literature and verify it with their own documented search for the specific question at hand — surveys age, and their scope may differ from yours.

2.5 The contribution test

Not every gap is worth filling. A gap qualifies as a doctoral contribution when it passes four tests:

1. Falsifiability. It can be stated as a proposition that some feasible observation would refute. "Little is known about handoffs in incident response" is a topic. "Receiving responders given a structured handoff note make fewer comprehension-level errors than those given free-text notes, under incomplete telemetry" is falsifiable.
2. Consequence. If the proposition were shown true or false, some belief or practice would change. Ask who would decide differently.
3. Novelty relative to the documented search. The search protocol shows the proposition has not been tested at the required level in this setting.
4. Feasibility under Chapter 1's constraints. The data, authorization and measures exist or can be obtained ethically within the program.

A useful drafting format is: "Current evidence supports A at level X in setting S. It does not establish B in setting T because [absent/weak/conflicting/untransferred evidence]. If B holds, practitioners would [decision]; if it does not, they would [other decision]. B can be tested by [design] and would be refuted by [observation]."

2.6 Worked example: several primary studies of decision support during incident triage

The real studies below are summarized from their published abstracts and pages linked; the synthetic studies are invented and labeled.

A candidate's search on decision support during triage yields, among others:

In plain terms: Read across the rows: the real studies describe context and trust, while the synthetic quantitative studies estimate effects in narrow settings; nobody has yet connected the two.

Synthesis. Real qualitative work (R1–R3) consistently points to context, trust, workload and organizational factors as determinants of triage quality. Experimental evidence about decision aids (S1) is lab-only with non-expert participants. Field evidence (S2) measures speed, not correctness, and is conflicted. Evidence specific to handoffs (S3) is qualitative and single-site. The candidate's gap statement:

"Current evidence describes, at qualitative levels and in single sites, that context is lost at incident handoffs (S3) and that context determines how analysts judge alerts (R1). No study in our documented search tests whether a structured handoff artifact changes the correctness of receiving responders' decisions, as opposed to their speed, under incomplete telemetry. If it does, CSIRTs would adopt and standardize such artifacts; if it does not, effort should shift elsewhere (for example, to telemetry or staffing). This can be tested by a controlled simulation with responders as participants, scored by blinded expert raters, and would be refuted by no difference — or a difference favoring free-text notes — in comprehension- and projection-level errors."

This statement passes the four tests: it is falsifiable, consequential, novel relative to the documented search and feasible with simulation (Chapter 5). It also names a theoretical discriminator (error type), drawn from the situation-awareness account.

2.7 Practical activity: build a literature map

A literature map is a matrix with claims as rows and studies as columns. Each cell records whether the study supports, contradicts or does not address the claim, at which evidence level, and with which main limitation. Unlike a citation list, a map shows the shape of evidence. The following offline script builds a simple text map from a synthetic extraction table and flags claims supported only below a chosen level. It uses only the standard library.

"""Chapter 2 literature-map helper. Study entries S1-S3 are synthetic; R1-R3 summarize published abstracts."""
EXTRACTION = [
    # (study, claim, direction, level)
    ("R1", "context-determines-alert-judgment", "supports", 3),
    ("R2", "manager-analyst-misalignment", "supports", 3),
    ("R3", "organizational-cycles-drive-performance", "supports", 3),
    ("S1", "aid-improves-correctness", "supports", 4),
    ("S2", "aid-improves-speed", "supports", 2),
    ("S3", "handoff-loses-context", "supports", 3),
    ("S1", "aid-improves-speed", "not-addressed", 0),
]
FIELD_THRESHOLD = 4  # a claim needs at least one level-4 field study to count as established

def build_map(rows):
    claims = {}
    for study, claim, direction, level in rows:
        claims.setdefault(claim, []).append((study, direction, level))
    return claims

if __name__ == "__main__":
    for claim, entries in sorted(build_map(EXTRACTION).items()):
        best = max((lvl for s, d, lvl in entries if d == "supports"), default=0)
        status = "established" if best >= FIELD_THRESHOLD else "open"
        cells = ", ".join(f"{s}:{d}:L{l}" for s, d, l in entries)
        print(f"{claim:42s} best=L{best} {status:11s} [{cells}]")

Expected output:

aid-improves-correctness                   best=L4 established [S1:supports:L4]
aid-improves-speed                         best=L2 open        [S2:supports:L2, S1:not-addressed:L0]
context-determines-alert-judgment          best=L3 open        [R1:supports:L3]
handoff-loses-context                      best=L3 open        [S3:supports:L3]
manager-analyst-misalignment               best=L3 open        [R2:supports:L3]
organizational-cycles-drive-performance    best=L3 open        [R3:supports:L3]

The output deliberately exposes a flaw in the script's rule: it marks "aid-improves-correctness" as established because S1 is a level-4 study — but S1 is a lab study with students, and the script's threshold was described as requiring a field study. A map is only as good as its coding scheme. Exercise EX012 asks you to fix this by adding a setting attribute and re-running.

2.8 From map to frontier

The research frontier is not "everything not yet studied." It is the set of claims that (a) matter for decisions, (b) are theoretically contested or unexamined, and (c) can now be tested because of new data, methods or access. A frontier statement for a thesis typically names two or three adjacent open claims and explains why the chosen one is the right first step — usually because its answer changes how the others should be studied.

For the Handoff Study, the adjacent claims are: whether handoff artifacts change correctness (chosen); whether effects depend on telemetry completeness (deferred to Chapter 6 as a boundary condition); and whether artifacts change long-term team learning (out of scope — would require longitudinal field access).

2.9 Decision science under uncertainty: a deeper reading

Section 2.3 introduced the theoretical accounts briefly so that the literature map could be built. A doctoral candidate needs more: enough command of each tradition to know what it actually claims, what its evidence base looks like, and where it is routinely misapplied to security work. This section revisits the decision-science traditions in that spirit. It is not a survey of psychology; it is a reading guide for an incident-response researcher who must use these theories as sources of testable predictions.

2.9.1 Situation awareness as a measurable construct

Endsley's theory paper is frequently cited in security; her companion measurement paper is cited far less, and that asymmetry matters. In Measurement of Situation Awareness in Dynamic Systems (Endsley, 1995) she discusses how situation awareness can be measured, including a freeze-probe approach in which a simulation is paused and the operator is asked questions about the current and projected state of the situation. The method that emerged from this line of work is widely known as SAGAT. Three features of the approach are directly relevant to incident-response research.

First, measurement is tied to a task analysis. The probe questions are not generic ("how aware are you?") but derived from the information requirements of the particular role. For a receiving responder after a handoff, the requirements would include which hosts are known compromised, which hypotheses are already excluded, which telemetry sources are unavailable, and what the adversary is expected to do next. A researcher who skips the requirements analysis and writes probe questions from intuition is measuring their own model of the task, not the responder's.

Second, the approach distinguishes levels. Because perception, comprehension and projection are probed separately, the researcher can test the mechanism claimed in Chapter 3 rather than only the net effect. That is the reason the Handoff Study separates perception errors from comprehension-and-projection errors: a handoff aid that improves the latter without the former is evidence for the mechanism, while an aid that improves only perception suggests a different mechanism (for example, that the template simply lists more facts).

Third, freeze-probe measurement is intrusive. Pausing a scenario can itself change behavior: participants may anticipate probes and rehearse facts they would otherwise ignore. Endsley discusses this concern; a study that uses freeze probes should report how often and when they occurred and should check whether performance after probes differs from performance before them. In a within-subject design, the probe schedule must be identical across conditions, or the probe becomes a confound.

The practical lesson for incident-response research is that "situation awareness" is not a feeling to be rated on a seven-point scale at the end of an exercise. Self-ratings of awareness exist and have uses, but they answer a different question (how confident does the responder feel?) and are known in the human-factors literature to diverge from probe-based measures. A thesis that reports only self-rated awareness should not claim to have measured awareness in Endsley's sense.

2.9.2 Recognition-primed decisions

The naturalistic decision-making tradition began with field studies of experienced decision makers working under time pressure. In Rapid Decision Making on the Fire Ground (Klein, Calderwood and Clinton-Cirocco, 1986), the authors reported interviews with fireground commanders and found that commanders rarely described comparing multiple options against one another. Instead they described recognizing the situation as a type, which suggested a plausible course of action, which they then evaluated by mentally simulating it — and modified or replaced only if the simulation exposed a problem. This pattern became known as the recognition-primed decision (RPD) model. Klein (2008) later summarized the naturalistic decision-making program as studying how people actually make decisions in demanding real-world settings rather than how they should decide under idealized assumptions.

For incident-response research, RPD changes what a decision aid should be expected to do.

• Aids that present option lists may be ignored by experts. If an experienced responder recognizes the situation and generates one workable action, a tool that presents five ranked options may be used to confirm the action already chosen rather than to choose among options. A study that measures whether the "top recommendation" was followed will then overestimate the tool's influence.
• Aids that support recognition and mental simulation may help most. Information that sharpens the recognition step (what kind of incident is this?) or that supplies inputs to the simulation step (what will happen if we isolate this host?) fits the way experts are described as deciding. The SHA's "expectations" field is designed with this in mind: it hands the receiver the sender's projection, which the receiver can then simulate against.
• Novices may decide differently. RPD describes experienced decision makers. Students and junior analysts, who lack a repertoire of situation types, may compare options more deliberately or may adopt the first option seen. A study that pools novices and experts may average two different decision processes and find a muddy effect that describes neither. This is a reason to stratify by experience (Chapter 5 §5.2) and to pre-specify experience as a moderator.

RPD also implies a caution about research methods. The original evidence was based on interviews after the fact. Retrospective reports are vulnerable to reconstruction and to the hindsight bias discussed in Chapter 1. Critical-decision interviewing techniques attempt to reduce this by anchoring the interview on specific moments and cues, but they cannot eliminate it. When an incident-response study borrows these methods, it should state that the decision process it reports is the process as described, not as observed.

2.9.3 Heuristics, biases and the conditions for trusting intuition

The heuristics-and-biases tradition, introduced to a broad audience in Tversky and Kahneman (1974), described judgmental heuristics — representativeness, availability, and adjustment from an anchor — that are often useful but lead to systematic errors in identifiable conditions. The paper predates computer incident response, but each heuristic has an obvious response analogue: judging an alert as malicious because it resembles a remembered intrusion (representativeness), estimating the likelihood of ransomware from recent headlines (availability), and failing to revise an initial scope estimate as evidence accumulates (anchoring).

A doctoral researcher should resist two tempting moves. The first is to diagnose biases from outcomes: "the team anchored on the first hypothesis" is often written after an incident ended badly, but the same persistence would be called "focus" if the first hypothesis had been right. Without a record of the evidence available at each decision point, an anchoring diagnosis is a hindsight judgment. The second move is to assume laboratory effect sizes transfer to practice. Heuristics research largely used controlled tasks with defined probabilities; incident work involves ambiguous cues, collaboration and tools. The direction of a bias may carry over while its magnitude does not.

The productive synthesis is the one already cited in §2.3.2: Kahneman and Klein (2009) agreed that intuitive judgment deserves trust when the environment is regular enough to be learned and the decision maker has had adequate practice with feedback. Incident response is not one environment but many. Disposition of a high-volume, well-understood alert type with confirmed outcomes may meet both conditions; scoping a novel intrusion in an unfamiliar network meets neither. A research design can operationalize this distinction by classifying scenarios by feedback quality (how often, and how quickly, would a real responder learn whether such a judgment was right?) and testing whether a decision aid's effect is larger where feedback is poor. That is a sharper prediction than "aids help," and it can fail.

2.9.4 Sensemaking and the collapse of shared frames

Weick's analysis of the Mann Gulch fire, The Collapse of Sensemaking in Organizations (Weick, 1993), examined how a smokejumper crew's shared understanding of their situation and their roles broke down as conditions changed faster than the crew could reinterpret them. The analysis is a case study and should be cited as such: it generates concepts, not effect sizes. Its value for incident-response research is a vocabulary for phenomena that the situation-awareness framing can miss.

• Sensemaking is social. Understanding is constructed in interaction — in what people say to each other, which interpretations get voiced and which are suppressed. A handoff is precisely such an interaction. A study that measures only the receiver's individual awareness may miss whether the handoff conversation surfaced disagreements.
• Role structure supports meaning. Weick argued that when the crew's role structure dissolved, so did the shared basis for interpreting events. In response, role ambiguity (who is incident lead after the shift change?) may degrade understanding as much as missing facts. A handoff aid that transfers facts but leaves role transfer implicit could leave this failure untouched.
• Commitment to a frame can be dangerous when the frame is wrong. Dropping a familiar interpretation is costly; teams tend to hold it until evidence is overwhelming.

These ideas generate testable predictions. For example, if role ambiguity is a separate pathway, then adding an explicit "decision authority" field to the SHA should reduce errors of omission (actions nobody took because everybody assumed someone else would) even if it does not change comprehension scores. A thesis can treat sensemaking as a rival account of the same phenomenon and design a measure that distinguishes them.

2.9.5 Drift toward boundaries

Rasmussen (1997) argued that risk management in a dynamic society is a control problem across levels — from government and regulators through companies and management to staff and work — and that work practices tend to migrate under pressures of efficiency and least effort toward the boundary of acceptable performance. Safety, in this view, depends on making boundaries visible and on feedback across levels, not only on rules at the operational level.

For incident response, Rasmussen's framing directs attention to decisions far from the console: staffing levels, alert-volume targets, outsourcing contracts, and reporting obligations. A response team under persistent pressure to close tickets quickly may migrate toward closing ambiguous cases as benign — not because any individual decides to accept risk, but because the local gradient rewards it. The Handoff Study is a micro-level intervention; Rasmussen's framing reminds the researcher that its effect may be swamped or reversed by pressures at higher levels. A transportability analysis (Chapter 6) that ignores management metrics is incomplete.

2.9.6 What the traditions jointly imply for design

In plain terms: Each tradition looks at a different level — one person, an expert, a judgment, a group, a whole system — and each makes a different, testable prediction about the handoff aid. The bottom row of misuses is a checklist for reviewing your own draft.

2.10 Incident-response teams and crisis coordination as research objects

2.10.1 What empirical work on response teams shows

Direct empirical research on incident-response teams remains thinner than research on detection. Ahmad, Hadgkiss and Ruighaver (2012) studied incident response teams in organizational settings and discussed challenges in how response supports the wider security function, including the tendency for response to concentrate on technical recovery while organizational learning from incidents receives less attention. Together with the SOC fieldwork already cited in §2.3.3 and the interview evidence of Kokulu et al. (2019), it supports a cautious generalization: in the organizations studied, the "after" phases of response — learning, feedback to prevention, and dissemination — were weaker than the "during" phases. This is consistent with NIST SP 800-61r3's decision to place improvement (ID.IM) inside the CSF's continuous functions rather than at the end of a linear life cycle.

For a doctoral literature review, the important feature of this body of work is its method profile: case studies, interviews and fieldwork in a small number of organizations, usually in one country and one sector, often without a comparison condition. That profile is appropriate for building concepts and inappropriate for estimating prevalence. A review that writes "studies show that most organizations neglect lessons learned" has converted a recurring qualitative theme into a population statement. The correct sentence names the method and setting: "interview and case studies in several organizations repeatedly report…"

2.10.2 The incident command system as a coordination template

Large incidents involve coordination across teams and organizations: internal IT, legal, communications, executives, outside forensic firms, insurers, law enforcement, regulators and, sometimes, other victims. Public-safety disciplines developed the incident command system (ICS) for multi-agency emergencies. Bigley and Roberts (2001) studied ICS in a fire-department setting and analyzed it as a form of high-reliability organizing for complex and volatile task environments. Their analysis emphasizes that ICS combines a formal, bureaucratic structure with mechanisms that allow that structure to change quickly — roles can be activated, merged or reassigned as the incident evolves — and with practices for building and maintaining shared understanding of the situation.

Two cautions apply when security programs adopt ICS vocabulary. First, an organizational chart with ICS titles is not ICS in the sense Bigley and Roberts analyze; the flexibility mechanisms and the training that makes rapid restructuring safe are the substance. Second, fire departments respond to physically visible incidents with shared professional norms; cyber incidents are often ambiguous in scope for days and involve parties with conflicting legal interests. Whether ICS-style structures improve cyber-incident coordination is an empirical question, not a settled fact. A thesis that proposes them should treat that as a hypothesis.

2.10.3 Cognition, communication, coordination and control

Comfort (2007), writing on crisis management in public administration, organized the problem around four linked capacities: cognition (recognizing the degree of emerging risk), communication, coordination and control. The framing is useful for incident-response researchers because it locates failures between organizations as well as within them. A cyber incident affecting a regional health network may involve a hospital, its managed-service provider, a software vendor, a national CSIRT and a regulator. Each has partial information; none controls the whole response. Delays and errors often arise at the interfaces: information that one party holds but cannot share under its contract or classification rules, or a decision that requires authority no single party has.

The Handoff Study as written examines handoffs between shifts in one team. The same mechanism — loss of hypotheses, gaps and expectations at a boundary — plausibly operates at organizational boundaries, with additional distortions introduced by information-sharing constraints such as TLP 2.0 designations, legal privilege and contractual confidentiality. That is a legitimate direction for future work (Chapter 8 §8.5), and it is also a reason to be careful about transporting a within-team result to cross-organization coordination.

2.11 Automation, AI assistance and the reliance problem

2.11.1 Use, misuse, disuse and abuse

Much current interest in response research concerns automated triage, summarization and recommendation. The human-factors literature on automation provides the conceptual tools to study this well. Parasuraman and Riley (1997) distinguished use of automation from misuse (over-reliance), disuse (under-reliance, such as ignoring or disabling it) and abuse (automating functions without due regard for the consequences for human performance). Each maps onto familiar response phenomena: analysts accepting an automated "benign" verdict without checking (misuse), analysts muting a noisy detection rule (disuse), and organizations automating containment actions whose failure modes no human is positioned to catch (abuse).

Lee and See (2004) reviewed research on trust in automation and argued that the design goal is appropriate reliance — trust calibrated to the automation's actual capabilities in context — rather than maximal trust. Parasuraman and Manzey (2010) reviewed complacency and automation bias and proposed an integrated, attention-based account of how both arise in human use of automation.

2.11.2 Research design consequences for AI-assisted response

These ideas sharpen the pitfalls already listed in Chapter 1 from Arp et al. (2022):

1. Accuracy of the aid is not the outcome. A triage model's precision and recall describe the aid; the research question concerns the decisions of the human–aid team. A study that reports only model metrics has not studied response.
2. Measure reliance directly. To test calibration, the study must include cases where the aid is wrong. If the aid is correct on every scenario, agreement with the aid cannot be distinguished from good judgment. A well-designed evaluation deliberately includes a pre-specified fraction of aid errors and measures whether participants detect them.
3. Separate conditions in which the aid is right from those in which it is wrong. The aggregate effect of an aid can be positive while it causes serious harm on its error cases. Report outcomes stratified by aid correctness.
4. Expect learning over time. Reliance changes with exposure. A one-session laboratory study measures first impressions; operational reliance after months may differ. State the time horizon of the claim.
5. Audit leakage before reporting accuracy. Kapoor and Narayanan (2023) documented data leakage as a widespread source of irreproducible and overoptimistic results across fields that use machine learning, and proposed a taxonomy of leakage types. In incident data, the commonest forms are temporal (training on cases that postdate test cases), duplicate (near-identical alerts in both splits) and label leakage (features computed from analyst notes that encode the disposition).
6. Large language models add their own pitfalls. When an assistant generates incident summaries, the outputs vary across runs and model versions, may contain fluent but unsupported statements, and may have been exposed during training to public material similar to the evaluation cases. A study should fix and record the model version and generation settings, run multiple samples, have raters score factual support against the source record, and treat public benchmarks as possibly contaminated. These points follow from general reproducibility requirements; they do not depend on any particular vendor's claims.

2.11.3 A testable reliance prediction for the Handoff Study

Suppose a later version of the SHA is pre-filled by a summarization model. The reliance literature predicts that receivers will check pre-filled fields less carefully than fields written by a colleague whose reliability they know. A discriminating design would compare (a) human-written SHA, (b) model-pre-filled SHA with no errors, and (c) model-pre-filled SHA with one seeded error per scenario in the "ruled-out hypotheses" field, measuring whether the error propagates into the receiver's decision. Condition (c) is ethically acceptable in simulation and is the only way to estimate the cost of misuse.

2.12 Measuring response quality: what the metrics literature warns

Operational response programs report metrics such as mean time to detect, mean time to respond, ticket counts and SLA compliance. Research needs metrics with known validity. Pendleton et al. (2016) surveyed systems security metrics and organized them around vulnerabilities, defenses, threats and situations; their survey is useful for showing how varied and loosely defined such metrics are. Herley and van Oorschot (2017) examined security as a scientific pursuit and criticized, among other things, claims that cannot be tested and the gap between security claims and the observations that could support them.

From these sources and from the measurement discussion in Chapter 1, five warnings follow for incident-response metrics.

1. Time metrics are censored and gamed. Time to contain depends on when detection occurred, which depends on attacker behavior and on the detection pipeline. Incidents still open at the end of a reporting window are censored (Chapter 1's lab). Teams under time targets can close tickets early. A time metric needs a precise start event, end event, and a rule for open cases.
2. Counts confound workload with performance. More incidents handled may mean better throughput or worse prevention.
3. Outcome metrics confound decision quality with luck. A contained incident may reflect good decisions or a clumsy adversary (§1.2.4).
4. Proxies drift. A metric that was a good proxy when introduced loses validity once people optimize it.
5. Aggregates hide distribution. Mean time to respond can improve while the worst cases worsen. Report distributions and tails.

The Handoff Study's choice of rater-scored decision errors against pre-defined defensible options is a response to these warnings: it measures the decision directly, at a known moment, against a standard fixed before the outcome is known. Its cost is ecological validity, which Chapter 6 addresses.

Common errors and limits

• Calling a review "systematic" without a recorded, repeatable search protocol and screening record.
• Treating standards' recommendations or vendor percentages as tested empirical findings.
• Mapping topics instead of claims; the map then cannot show contradictions.
• Declaring absence of evidence after one database query.
• Stating a gap as a topic ("handoffs are understudied") rather than a falsifiable proposition.
• Choosing a theory after the results to explain them (Chapter 1's HARKing) rather than using competing theories to design a discriminating test.
• Limit: this chapter's summaries of published studies rely on their abstracts and public pages; a real review must read and appraise full texts.

Exercises — chapter 2

EX008 — Search protocol. Objective: write a repeatable search protocol. Inputs: the Handoff Study question. Constraints: at least three sources of different types; explicit eligibility criteria; stopping rule. Steps: (1) state the question in population–intervention–comparison–outcome–context form; (2) write search strings for two databases; (3) define eligibility and exclusion reasons; (4) define screening and agreement procedure; (5) define a stopping rule. Expected evidence: versioned protocol (≤ 2 pages). Rubric: 3 = repeatable by another researcher, criteria unambiguous, agreement measured; 2 = one element vague; 1 = keyword list only; 0 = none.

EX009 — Appraisal form application. Objective: appraise a study for the claim it can support. Inputs: synthetic study S2 (§2.6). Constraints: use the appraisal elements in §2.2.3. Steps: complete the form; assign an evidence level; list the three most serious threats; state the strongest claim S2 can support. Expected evidence: completed form and one-sentence claim. Rubric: 3 = threats correctly identified (vendor conflict, speed-only, concurrent change, single site), claim properly limited; 2 = one threat missed; 1 = accepts 35% figure as effect; 0 = no appraisal.

EX010 — Competing predictions. Objective: derive discriminating predictions from theory. Inputs: the five accounts in §2.3 and a proposed intervention of your choice (not the handoff note), e.g., automatic enrichment of alerts with asset-owner data. Constraints: each prediction must name an observable. Steps: build a table like §2.3.6; identify the one observation that would best discriminate between the two most plausible accounts. Expected evidence: table and a paragraph. Rubric: 3 = predictions differ and are observable, discriminator well chosen; 2 = predictions overlap; 1 = generic; 0 = no theory link.

EX011 — Classify the evidence gap. Objective: distinguish absent, weak, conflicting and untransferred evidence. Inputs: four synthetic statements: (a) "Two large field studies disagree on whether alert-volume caps reduce misses"; (b) "Checklists reduced errors in surgical handoffs; no security studies exist"; (c) "Three vendor reports claim AI triage halves response time"; (d) "No study in our documented search measured responder fatigue across 24-hour incidents." Constraints: justify each classification and propose the next research step. Steps: (1) restate each candidate gap as a question; (2) list the evidence that exists for it; (3) assign one gap type and justify it in one sentence; (4) name the study that would close it. Expected evidence: four short paragraphs. Rubric: 3 = all classified correctly with appropriate next steps; 2 = one misclassified; 1 = two or more; 0 = none.

EX012 — Repair the literature map. Objective: make a coding scheme match its stated rule. Inputs: the script in §2.7. Constraints: offline; keep all original rows. Steps: (1) add a setting field ("lab" or "field") to each row (R1–R3, S2, S3 field; S1 lab); (2) change the rule so a claim is established only with a level-4 field study; (3) run and record the output; (4) explain in 100 words what changed and why it matters. Expected evidence: modified code, exact output, explanation. Rubric: 3 = correct code, exact output with no claim established, clear explanation; 2 = output correct, explanation thin; 1 = rule still wrong; 0 = not run.

EX013 — Falsifiable gap statement. Objective: convert a topic into a contribution. Inputs: the topic "incident communication with executives during ransomware." Constraints: use the drafting format in §2.5; the refuting observation must be feasible. Steps: write the statement; test it against the four contribution tests; revise once. Expected evidence: draft, test notes, revision. Rubric: 3 = passes all four tests with explicit refutation; 2 = one test weak; 1 = still a topic; 0 = none.

EX014 — Frontier memo. Objective: justify a first study among adjacent open claims. Inputs: your EX013 statement and two adjacent open claims you identify. Constraints: 400–600 words; cite only sources you have read. Steps: describe the three claims; explain dependencies; justify the first; state what is out of scope and why. Expected evidence: memo. Rubric: 3 = dependencies clear, choice justified by how its answer shapes the others, scope honest; 2 = justification by convenience only; 1 = no dependency analysis; 0 = none.

Chapter synthesis

A research frontier is found, not asserted. It emerges from a documented search, an appraisal that reads each study for the claim it can bear, a map of claims rather than topics, and competing theories that predict different outcomes. The output is a falsifiable gap statement that would change decisions if answered. Chapter 3 turns the Handoff Study's gap statement into a specific contribution with a mechanism, hypotheses, a baseline and predeclared failure conditions.

Continue studying — chapter 02 of 08

← Chapter 01 · Course chapter index · Chapter 03 →

Continue to the next authored lesson.