Chapter 1 — Advanced research integrity and methods

Lesson map

Lesson 01 published. Map, example, EX001.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["1.1 Incident response as an object of research"]
  N2["1.2 The epistemology of incident evidence"]
  N3["1.3 Research integrity applied to incident-response"]
  N4["1.4 Reproducibility and open science under constrain"]
  N5["1.5 Ethics and authorization for studying real incid"]
  N6["1.6 Measurement validity for response quality"]
  N7["1.7 Common failure modes in incident-response resear"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX001"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["1.1 Incident response as an object of research"]
  N2["1.2 The epistemology of incident evidence"]
  N3["1.3 Research integrity applied to incident-response"]
  N4["1.4 Reproducibility and open science under constrain"]
  N5["1.5 Ethics and authorization for studying real incid"]
  N6["1.6 Measurement validity for response quality"]
  N7["1.7 Common failure modes in incident-response resear"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX001"]

[Learning outcomes and connection]
    |
    v
[1.1 Incident response as an object of research]
    |
    v
[1.2 The epistemology of incident evidence]
    |
    v
[1.3 Research integrity applied to incident-response]
    |
    v
[1.4 Reproducibility and open science under constrain]
    |
    v
[1.5 Ethics and authorization for studying real incid]
    |
    v
[1.6 Measurement validity for response quality]
    |
    v
[1.7 Common failure modes in incident-response resear]
    |
    v
[Worked example] --> [Exercise EX001]

Learning outcomes and connection

By the end of this chapter the learner will be able to:

1. Classify any statement in an incident record or a research paper as an observation, an inference or a claim, and state what would be needed to promote it to the next level.
2. Explain why incident evidence is unusual as research data: it is produced by systems not designed for research, altered by the response itself, read with hindsight, and usually encumbered by privacy, legal and contractual restrictions.
3. Apply integrity safeguards (pre-specification, deviation logs, separation of exploratory and confirmatory work, conflict-of-interest disclosure) to an incident-response study.
4. Design a tiered reproducibility plan that releases as much as can lawfully and safely be released, and say honestly what cannot be checked.
5. Identify the stakeholders, authorizations and harms involved in studying real incidents, using the Menlo Report principles as a working structure.
6. Choose and defend measures of response quality, recognizing threats to construct validity, censoring in time-based metrics, base-rate effects and rater disagreement.
7. Write a research protocol that a hostile but fair reviewer could audit, including the conditions under which the study would be stopped or its claim abandoned.

The chapter is the methodological contract for the rest of the book. Chapter 2 will use its claim vocabulary to read the literature; Chapter 3 will use its protocol template; Chapters 4–6 will use its measurement and validity tools; Chapters 7–8 will use its integrity and reproducibility standards to judge the finished work.

1.1 Incident response as an object of research

Practitioners learn incident response as a set of activities. The current NIST guidance, SP 800-61 Rev. 3 (April 2025, superseding Rev. 2 of 2012), frames those activities as part of cybersecurity risk management organized by the six CSF 2.0 Functions — Govern, Identify, Protect, Detect, Respond and Recover — and places continuous improvement in the Identify Function's Improvement category (ID.IM). Its full text notes that preparation activities are broader risk-management work that supports response, while Detect, Respond and Recover form the response itself, and that lessons learned should often be shared as soon as they are identified rather than after recovery ends. Recovery planning and testing are treated in SP 800-184 (December 2016), which emphasizes prioritizing resources, realistic test scenarios and learning from past events, including other organizations' events.

A researcher needs a different stance. The question is not "what should a responder do next?" but "what do we have warrant to believe about how response works, and how confident should we be?" That shift has five consequences.

First, the unit of analysis must be chosen, not assumed. A study can be about an incident (a bounded case), a decision within an incident (for example, the choice to isolate a host), an actor (a responder, a team, a vendor), an artifact (a playbook, a handoff note, a detection rule), an organization or a population of organizations. Many weak claims in the field come from measuring at one level and concluding at another: counting tickets (an artifact level) and concluding about team capability (an organizational level), or interviewing three analysts and concluding about "SOCs."

Second, incidents are rarely repeatable. A chemist can rerun a reaction; an incident happens once, in a particular organization, against a particular adversary, with particular staff on shift. Researchers approximate repetition in three ways: by studying many incidents (with all the heterogeneity that brings), by simulating incidents in exercises or laboratories (with the realism gap that brings), or by studying components that do repeat, such as triage decisions on a stream of alerts. Each approach buys some control at the price of some relevance; none is free.

Third, the response changes the evidence. Containment, reimaging, password resets and log rotation alter or destroy the records a later analyst would want. The decision of what to collect is itself a response decision. A research dataset built from incident records is therefore not a passive sample of what happened; it is a sample of what the responders chose and were able to preserve.

Fourth, outcomes are adversarial and partly unobservable. An undetected intrusion produces no incident record. A "successful" containment might reflect a capable team or an adversary who had already achieved its goal and left. Ground truth is often unavailable, delayed or contested.

Fifth, the work is sociotechnical. Response quality depends on tools, telemetry, procedures, authority, fatigue, communication and organizational incentives. Fieldwork in security operations centers illustrates this: an anthropological study of a corporate SOC over six months modeled analyst burnout as a human-capital problem arising from cycles among human, technical and managerial factors, and an interview study of 18 SOC analysts and managers reported disagreements between managers and analysts that could undermine effectiveness. A survey of SOC literature in IEEE Access (2020) concluded that research tends to address people and technology while neglecting the processes, especially non-technical ones, that connect them. A study that measures only the tool will miss most of what determines response quality.

These consequences do not make rigorous research impossible. They make it necessary to be explicit about what each study can and cannot show.

1.2 The epistemology of incident evidence

1.2.1 What counts as evidence

SP 800-61 Rev. 3 adopts a broad definition: under subcategory RS.AN-07, it notes that formal chain-of-custody handling may not be performed for every incident, but that collected incident data is still evidence, "grounds for belief or disbelief; data on which to base proof or to establish truth or falsehood," a definition it attributes to NIST SP 800-160 Vol. 1 (SP 800-61r3 PDF, Table 3). The same table asks organizations to preserve the integrity and provenance of incident data (RS.AN-07) and of the records of actions taken during an investigation (RS.AN-06), and warns that incident response records can contain sensitive information about exploited vulnerabilities, breaches and users.

For research, the useful part of this definition is that evidence is defined by its role in belief, not by its form. A log line, a ticket comment, a screenshot, an interview transcript and a responder's memory can all be evidence. What differs is how much they can support and how their support can fail.

Every piece of incident evidence has at least these properties, and a research dataset should record them explicitly:

Lesson table; scroll horizontally to read all columns →

The technical book in this series develops the operational side of these properties (producer clocks, collection receipts and derived views). At doctoral level the point is that each property is also a validity threat to any claim built on the data.

1.2.2 Observation, inference and claim

The most important habit in incident-response research is to keep three kinds of statement apart.

• An observation is what a named source recorded, stated with its producer and time: "The VPN concentrator log exported at 14:02 UTC contains a successful authentication for account u-417 from address 192.0.2.10 at 03:11 UTC."
• An inference is a conclusion drawn from one or more observations under stated assumptions: "Assuming the concentrator clock was within one minute of UTC and the account is not shared, the account holder or someone with their credentials authenticated at about 03:11."
• A claim is the statement a researcher or responder will defend to others, with an explicit scope: "In this incident, initial access most likely used valid credentials (moderate confidence; the alternative of a session-token replay is not excluded because token logs were unavailable)."

A research claim sits one level higher again: "Across the 41 incidents in our sample, receiving responders who were given a structured handoff note reached a correct containment decision more often than those who were not." Its warrant depends on every observation and inference beneath it, on the sampling of incidents and on the measurement of "correct."

Two disciplined practices follow. First, every claim should be traceable downward to the inferences and observations that support it, so that a reviewer can find exactly where an assumption entered. Second, every inference should name the observation that would defeat it. "The account was not shared" is a defeater-bearing assumption; if an identity-management record shows two people using u-417, the inference collapses. Writing the defeater before looking for it is a small act of pre-commitment that protects against confirmation bias.

The evidence ladder below is a teaching device used throughout this book; it is not a published standard.

Lesson table; scroll horizontally to read all columns →

In plain terms: Claims climb the ladder only when stronger evidence arrives, and they fall when that evidence is contradicted; the table tells you what would move each claim up or down.

1.2.3 Absence of evidence

Incident analysts constantly reason from absence: "no outbound traffic to the suspicious domain," "no further logins." Absence is evidence only when three conditions hold: the source covered the relevant asset and period, the source would have recorded the event if it had occurred (it was configured, running and not saturated), and the analyst searched correctly (right fields, right time zone, right identifiers). A research protocol that uses "no recurrence within 30 days" as an outcome inherits all three conditions. If telemetry coverage changed after the incident — as it often does when an organization adds monitoring during recovery — then recurrence detection is not comparable before and after.

1.2.4 Hindsight and outcome knowledge

Everyone who studies an incident after the fact knows how it ended. In experiments reported by Fischhoff (1975), receiving outcome knowledge increased the judged likelihood of the reported outcome and changed which details seemed relevant, while judges were largely unaware of that influence and overestimated what they — and others — would have known without it. In the safety-science tradition, Richard Cook's short treatise How Complex Systems Fail states that hindsight biases post-accident assessments of human performance, and argues that post-accident attribution to a single "root cause" is fundamentally wrong because overt failure requires multiple contributing faults.

That second point sits in visible tension with SP 800-61 Rev. 3, which recommends analyzing an incident "to find the underlying or systemic root causes" (RS.AN-03, R3, in the PDF). A researcher should not resolve the tension by picking a side; it should be operationalized. Note that NIST's wording is plural and includes "systemic." A study that codes "root cause" as a single categorical variable is making a modeling choice that Cook's argument would reject; a study that codes contributing factors as a set, with the evidence for each, is compatible with both sources. State the choice and justify it.

Hindsight affects research design in concrete ways:

• Judging decision quality by outcome (outcome bias). A containment decision that turned out well may have been poorly reasoned given what was known at the time; one that turned out badly may have been the best available choice. Chapter 5 separates decision quality given information available at decision time from outcome.
• Retrospective interviews. Responders interviewed after an incident reconstruct their reasoning with knowledge they did not have. Interview protocols should anchor questions to time-stamped artifacts ("At 10:40 the ticket says X; what else were you looking at then?").
• Coding of reports. Coders who read the end of a post-incident report before its timeline will code early decisions differently. Blind coders to outcomes where possible, or present information in time order with later sections withheld.

1.2.5 The response as an intervention on its own evidence

Because responders shape what is recorded, incident data have reactivity. Three patterns matter for research. Instrumentation drift: monitoring added after an incident increases detection of later events, making it look as if incidents rose. Selective preservation: well-resourced teams preserve more, so their incidents look more "complete" and better understood. Narrative convergence: post-incident reports are written to explain; contradictory early observations are often omitted from the final narrative. A researcher using reports as data should, where permitted, also obtain the contemporaneous working record (tickets, chat logs, action logs — the RS.AN-06 records) and treat the final report as one interpretation among others.

1.3 Research integrity applied to incident-response research

Research integrity is usually summarized as honesty, transparency and accountability in proposing, performing and reporting research. In incident response, the general principles meet specific temptations.

1.3.1 Questionable research practices and why IR is vulnerable

The broader scientific literature has documented how flexible analysis inflates false findings. Ioannidis (2005) argued, using a simple model, that the probability a published positive finding is true falls with small samples, small effects, many tested relationships, flexible designs and strong interests. Simmons, Nelson and Simonsohn (2011) showed through simulations and demonstrations that undisclosed flexibility in data collection and analysis can make false-positive results very likely. Gelman and Loken (2014) described how this can happen even without deliberate "fishing": when analysis choices depend on the data, the effective number of comparisons is larger than the one analysis reported (the "garden of forking paths"). Kerr (1998) named the practice of presenting a post hoc hypothesis as if it were a priori: HARKing, hypothesizing after the results are known.

Incident-response research has every risk factor these authors identify:

• Small samples. A single organization may experience a handful of major incidents per year.
• Many possible outcome measures. Time to detect, time to contain, time to recover, scope, cost, recurrence, analyst confidence, stakeholder satisfaction, each with several definitions.
• Flexible case definitions. What counts as one incident versus two? When does an incident start?
• Strong interests. Vendors evaluate their own products; teams evaluate their own processes; researchers depend on partner organizations who would prefer flattering results.
• Post hoc narratives. Incidents arrive as stories, and stories invite explanation after the fact.

Security has its own documented versions. Arp et al. (USENIX Security 2022) identified ten pitfalls in machine-learning-based security research — sampling bias, label inaccuracy, data snooping, spurious correlations, biased parameter selection, inappropriate baselines, inappropriate performance measures, base-rate fallacy, lab-only evaluation and inappropriate threat models — and reported them to be widespread in a study of 30 papers from top-tier security venues. Pendlebury et al. (USENIX Security 2019) argued that published malware-classification results were inflated by spatial bias (unrepresentative train/test distributions) and temporal bias (time splits that would be impossible in deployment). Sommer and Paxson (2010) explained why machine-learning approaches to network intrusion detection struggle outside the "closed world" of their training assumptions. Although these papers focus on detection, their pitfalls transfer directly to research on triage tools, automated response and decision support.

1.3.2 Pre-specification and preregistration

Nosek et al. (2018) describe preregistration as specifying the research plan in advance, before seeing the outcome data, so that confirmatory (hypothesis-testing) analyses can be distinguished from exploratory (hypothesis-generating) ones. The Registered Reports format goes further: peer review of the question and method happens before data collection, and a protocol that passes is provisionally accepted regardless of results, which removes the incentive to produce a positive story.

Three adaptations make pre-specification workable in incident-response research.

1. Pre-specify against data you have not yet seen, not against time. Field studies of incidents often begin with partial access to historical records. A preregistration is honest only if it is written before the researcher has examined the outcome variables in the data being used for confirmation. If you have already explored part of a dataset, preregister against a held-out part, or against incidents that have not yet occurred (prospective collection), and say so.
2. Pre-specify qualitative work differently. An interview or ethnographic study cannot specify its findings in advance, but it can specify its sampling logic, interview guide, coding procedure, the point at which coding categories will be frozen, and how disconfirming cases will be sought. "Exploratory" is a legitimate label; it is not a license to present exploratory patterns as confirmed.
3. Keep a deviation log. Real incident studies deviate: a partner withdraws, a log source turns out to be unusable, an incident type is reclassified. Every deviation should be recorded with date, reason, who decided, and whether it was made before or after seeing outcome data. A protocol with a transparent deviation log is more credible than one that claims to have gone perfectly.

1.3.3 Originality, attribution and provenance

At doctoral level, originality means that the contribution is new and that its relation to prior work is accurately represented. Incident-response research presents specific attribution problems:

• Practitioner knowledge without a citable source. Much of the field's knowledge lives in talks, blogs, vendor reports and internal wikis. When a practice is common but undocumented, say so ("widely reported by practitioners; we found no peer-reviewed evaluation") rather than either claiming it as your idea or attaching a citation that does not actually support it.
• Vendor and industry reports. Reports such as large annual breach reports compiled from vendors' and partners' caseloads can be valuable, but their samples reflect who engaged those organizations and what they chose to publish. Cite them for what they are — descriptive accounts of a particular caseload — and do not use their percentages as population estimates.
• Data provenance. Every dataset used in a thesis should have a provenance record: who created it, from which sources, under which permission, with which transformations, and which version was analyzed. "We used a public dataset" is insufficient if the dataset has several versions or was relabeled.
• AI-assisted writing and analysis. Where language models or automated tools generate text, code, labels or summaries, record what was generated, by which tool and version, and how it was checked. A generated citation must be verified against the actual source before it is used — fabricated references are an integrity failure regardless of how they were produced.

1.3.4 Conflicts of interest and partner pressure

Incident-response research often depends on partners: an organization that provides records, a vendor that provides a tool, a funder interested in a result. A conflict-of-interest statement should name each relationship and, more importantly, the protocol should structurally limit the conflict: pre-specified outcomes, independent coders blind to condition, publication rights agreed before data access, and an explicit statement of what the partner may review (accuracy and confidentiality, not conclusions). If a partner can veto publication of unfavorable results, the study cannot support a claim of effectiveness, and the thesis should say so.

1.4 Reproducibility and open science under constraint

1.4.1 Definitions

This book follows the definitions adopted by the U.S. National Academies of Sciences, Engineering, and Medicine in Reproducibility and Replicability in Science (2019): reproducibility is obtaining consistent results using the same input data, computational steps, methods, code and conditions of analysis (also called computational reproducibility); replicability is obtaining consistent results across studies aimed at the same scientific question, each of which has obtained its own data. The report notes that even a rigorous, transparent study may fail to replicate, because new data differ.

The distinction matters in incident response because the two are blocked by different obstacles. Reproducibility is blocked mainly by access: the original incident records usually cannot be shared. Replicability is blocked mainly by heterogeneity: no two organizations, incidents or adversaries are alike, so a second study is never a pure repetition. Chapter 4 addresses reproduction of a baseline; Chapter 6 addresses replication across contexts.

1.4.2 Open-science expectations in security venues

Open science is becoming a formal expectation in security research. The USENIX Security 2026 call for papers requires every paper to include a separate "Ethical Considerations" appendix and a separate "Open Science" appendix listing where the artifacts needed to evaluate the contribution are located; artifacts must be available at submission or their unavailability must be explained. The FAIR principles (Wilkinson et al., 2016) describe data stewardship goals — findable, accessible, interoperable and reusable — that apply to metadata even when the data themselves are restricted. Munafò et al. (2017) collect measures across methods, reporting, reproducibility, evaluation and incentives intended to improve the reliability of research.

1.4.3 The constraint taxonomy

Incident data are almost always restricted. A doctoral researcher should identify which constraint applies to which artifact, because each constraint permits different mitigations.

Lesson table; scroll horizontally to read all columns →

In plain terms: Each constraint removes some actions from the researcher's menu but rarely all of them; the last column shows what work remains possible inside the limit.

1.4.4 De-identification is not a magic word

Removing names does not make incident data anonymous. Incident records are high-dimensional and sparse: a combination of timestamps, rare software versions, internal hostnames and job roles can single out an organization or a person. Narayanan and Shmatikov (2008) demonstrated robust de-identification attacks on large sparse datasets using small amounts of auxiliary information. Sweeney (2002) formalized k-anonymity as a model in which each released record is indistinguishable from at least k−1 others on identifying attributes. Dwork, McSherry, Nissim and Smith (2006) introduced calibrating noise to the sensitivity of a query, the foundation of differential privacy, which bounds what a released statistic can reveal about any individual record. NIST's SP 800-188 (September 2023) describes de-identification techniques and governance for government datasets, with the goal of limiting disclosure risks to individuals and establishments while allowing meaningful statistical analysis.

Two lessons follow for incident-response researchers. First, organizations are data subjects too. An "anonymized" incident narrative that mentions a regional utility, a ransomware family and a date range may identify the organization to anyone in the sector. SP 800-188's explicit inclusion of establishments is a useful reminder. Second, de-identification is a governance process, not a transformation. Document the threat model (who might try to re-identify, with what auxiliary data), the technique, the residual risk and who approved release.

1.4.5 A tiered artifact plan

Because "release everything" is rarely possible and "release nothing" is rarely necessary, plan artifact release in tiers:

Lesson table; scroll horizontally to read all columns →

A strong thesis aims for T0–T2 at minimum and states, in its Open Science section, why higher tiers were not possible and what a reviewer can and cannot check. Synthetic data deserve their own caution: they validate the pipeline, not the finding. A model trained or evaluated only on synthetic incidents has demonstrated nothing about real incidents unless the synthetic generator itself has been validated against real distributions — and even then only for the properties checked.

1.5 Ethics and authorization for studying real incidents

1.5.1 Why incident research needs its own ethical analysis

Biomedical and behavioral research ethics grew around the 1979 Belmont Report's principles of respect for persons, beneficence and justice. The Menlo Report (August 2012; now marked by DHS as archived content) adapted those principles for information and communication technology research and added a fourth, respect for law and public interest. It proposes operationalizing them through stakeholder identification and informed consent; balancing risks and benefits; fairness and equity; and compliance, transparency and accountability, supported by ethical review boards and internal self-evaluation such as an Ethical Impact Assessment. The report argues that ICT research raises new challenges because of ubiquitously connected environments overlaid with varied and often discordant legal regimes. Although archived, its principles remain a clear structure for analysis; check current institutional and legal requirements separately.

Incident research raises problems that a generic human-subjects form does not anticipate:

• The people in the data did not volunteer. Employees whose accounts appear in incident records, customers whose data were exposed and third parties whose systems were used as infrastructure are all affected by research on those records.
• Responders are workers. Interviewing or observing responders creates risks of performance evaluation, blame and employment consequences, especially after a damaging incident. Consent obtained through a manager may not be free.
• Organizations can be harmed. Publication can reveal security weaknesses, attract regulators or litigants, or damage reputation, even when no individual is identified.
• Adversaries are stakeholders of a peculiar kind. Research may reveal detection gaps or response patterns useful to attackers. The ethical concern is not the adversary's interest but the downstream harm to defenders and the public.
• The research can interfere with response. Collecting extra data, interviewing during an active incident or delaying containment to observe would be an intervention with real costs.

1.5.2 Applying the four principles

Lesson table; scroll horizontally to read all columns →

In plain terms: The Menlo principles become concrete only when each one is turned into a question about the specific incident data and a safeguard that answers it.

1.5.3 Layers of authorization

Permission to study incident data comes in layers, and each must be obtained separately. A responder's operational access to records is not research authorization.

1. Legal basis. Does the jurisdiction permit processing these records for research? Data-protection law, sector rules, employment law and contractual terms may all apply. In the United States, federally supported human-subjects research falls under the Common Rule; other jurisdictions have their own regimes. This book does not provide legal advice; the protocol should record which legal review occurred and its conclusion.
2. Data controller or owner permission. The organization that holds the records must authorize the specific use, scope and duration, normally in a data-use agreement.
3. Ethics review. An independent research ethics board, where one exists, should review the protocol. If the researcher has no institutional board, the protocol should still include an Ethical Impact Assessment and an external reviewer, and the thesis should state the absence of formal review.
4. Participant consent. For interviews, surveys and observation, obtain informed consent from each participant, including the right to withdraw and clear statements about what will be published.
5. Operational coordination. Agree with the incident lead how research activity will be paused or abandoned if it conflicts with an active response. The response always takes priority.

1.5.4 Specific hazards and safeguards

Incidental discoveries. Researchers analyzing incident data may find a still-open vulnerability, an unreported compromise or evidence of wrongdoing. The protocol should state in advance to whom such findings are reported and how quickly, and that the researcher will not investigate further outside scope.

Dual use and release timing. A study showing that a class of detection rule fails under a specific condition is valuable to defenders and attackers alike. Consider coordinated disclosure to affected vendors, delayed release of technical specifics and publication of defensive guidance alongside findings.

Responder welfare. Fieldwork in SOCs has documented burnout and morale cycles (Sundaramurthy et al., SOUPS 2015). Asking exhausted responders to complete research instruments during or soon after a major incident adds burden. Offer timing control, keep instruments short and never make participation a condition of work.

Blame. Post-incident studies can become instruments for assigning fault. State in the consent material that individual-level findings will not be reported to management, and design analysis at the team or process level unless participants explicitly agree otherwise.

Stop rules. A protocol should list conditions that pause or end the study: an active incident requiring the data custodian's full attention, discovery of a legal hold, a participant complaint, evidence of re-identification risk exceeding the approved threshold, or a request by the data owner to withdraw.

1.6 Measurement validity for response quality

1.6.1 Constructs, operationalizations and the four validities

Most incident-response research ultimately wants to say something about quality: of a decision, a response, a team, a tool or a plan. Quality is a construct, an abstract property that cannot be observed directly. Researchers operationalize it through measures: time intervals, correctness against a reference answer, expert ratings, scope of impact, recurrence, cost. The quality of a study depends on how well the measures capture the construct.

Methods textbooks commonly organize threats into four families. This book uses that standard vocabulary without attaching it to a single source:

• Construct validity: do the measures represent the intended construct? (Does time-to-contain measure response quality, or only speed?)
• Internal validity: does the observed relationship reflect the proposed cause rather than a confounder? (Did the new playbook improve outcomes, or did it arrive at the same time as additional staff?)
• External validity: does the result hold for other settings, people, incidents and times? (Does a result from a university SOC hold in a hospital?)
• Statistical conclusion validity: are inferences about the size and uncertainty of an effect justified by the data and analysis? (Twelve incidents cannot establish a small effect.)

1.6.2 The trouble with time metrics

Time-based measures — mean time to detect, to respond, to contain, to recover — are attractive because they are numeric and appear objective. They carry at least five threats.

1. Definition ambiguity. "Detect" can mean the first alert, the first human acknowledgement, or the moment an analyst concluded an incident was real. "Contain" can mean the first containment action or the point at which no further adversary activity was observed. Two organizations reporting "mean time to contain" may be measuring different intervals.
2. Unknown start times. Intrusion start is often inferred retrospectively and revised as investigation proceeds. Time-to-detect therefore depends on how hard the team looked for the beginning.
3. Censoring. At the time a dataset is closed, some incidents are not yet contained. Excluding them biases averages downward, because the excluded incidents are, by definition, the slow ones. Section 1.10's lab demonstrates this.
4. Goal displacement. When a time metric becomes a performance target, teams can improve it by closing tickets early, splitting incidents or reclassifying them. The measure then tracks behavior around the target rather than the underlying quality.
5. Speed versus correctness. A fast containment that disrupts a critical service, or that alerts an adversary before scope is known, may be worse than a slower one. Time measures should be paired with correctness and harm measures.

1.6.3 Correctness and the ground-truth problem

Correctness requires a reference answer. In real incidents the reference is often unavailable; in simulations it is available because the designer wrote it, but then the study measures performance on the designer's scenario. Three strategies help.

• Expert panel reference. Multiple independent experts, blind to condition, judge what a good decision would have been given the information available at decision time. Their agreement must be measured.
• Designed scenarios with known ground truth. Synthetic or exercise scenarios in which the true state is known. The realism gap must be acknowledged, and SP 800-84's guidance on exercise design (CSRC page, September 2006) should be treated as a dated starting point for exercise structure, not a current evaluation standard.
• Multi-criteria rubrics. Score separate dimensions — scope accuracy, appropriateness of containment, preservation of evidence, communication, harm avoided — rather than a single overall grade.

Labels in real operational data are especially fragile. In a qualitative study of SOC practitioners, Alahmadi, Axon and Martinovic reported that most alarms practitioners considered false positives were attributed to benign triggers — true alarms explained by legitimate behavior in the environment — and argued that vendors and researchers need to distinguish types of false positive to evaluate tools properly. A study that treats analyst dispositions ("false positive") as ground truth is therefore measuring a mixture of detection error and environmental context. Arp et al.'s "label inaccuracy" pitfall names the same risk.

1.6.4 Rater agreement

When human judges score decisions or code reports, their agreement is part of measurement validity. Cohen's kappa (1960) corrects observed agreement for agreement expected by chance:

κ = (pₒ − pₑ) / (1 − pₑ)

where pₒ is observed proportion of agreement and pₑ is the agreement expected if the raters' category frequencies were independent. Landis and Koch (1977) proposed verbal benchmarks for kappa values that are widely quoted; treat any such benchmarks as conventions, not thresholds of truth. Low agreement means the construct is not being measured consistently: fix the rubric, train raters and re-measure before analyzing outcomes. Report agreement on the actual study data, not only on training items.

1.6.5 Base rates

Many incident-response tools are classifiers: they flag alerts, prioritize cases or recommend actions. The value of a flag to a responder depends on the positive predictive value (the proportion of flagged items that are truly positive), which depends heavily on the base rate. With 99% sensitivity and a 1% false-positive rate, a flag is right about 92% of the time when 10% of items are truly malicious, but only about 9% of the time when 0.1% are — the computation is shown in Section 1.10. Arp et al. list ignoring this effect as the base-rate fallacy. Any research claim about the operational usefulness of a triage tool must state the base rate in the evaluation data and in the intended deployment.

1.7 Common failure modes in incident-response research claims

The table below catalogs failure modes a doctoral reviewer should look for. Most have counterparts in the general methods literature; the examples are specific to incident response.

Lesson table; scroll horizontally to read all columns →

In plain terms: Most of these failures make results look better than they are; the right-hand column is the habit that prevents each one, and a protocol should show where each habit is applied.

1.8 Worked example: a proposed study using sensitive incident records and a public benchmark

All organizations, records and numbers in this example are synthetic.

A doctoral candidate, working with the fictional Northbridge Municipal Services CSIRT (NMS-CSIRT), proposes the following study:

"We will show that our triage-assistant model improves incident response by training it on five years of NMS-CSIRT incident tickets and testing it on the public OpenTriage-S benchmark, where it achieves higher accuracy than the published baseline."

A reviewer applying this chapter would decompose it.

Step 1 — Identify the claim and its scope. "Improves incident response" is a construct-level claim about real response. The proposed evidence is accuracy on a benchmark. The claim exceeds the evidence by at least two levels: from benchmark accuracy to triage decision quality, and from triage decision quality to response outcomes. A defensible restatement: "On OpenTriage-S, the model's top-1 triage category matches the benchmark label more often than the published baseline does, under a time-respecting split; we do not test effects on real response decisions."

Step 2 — Examine the data. NMS-CSIRT tickets contain employee names, citizen data in attachments, internal hostnames and descriptions of unremediated weaknesses. Ticket dispositions were entered by analysts under time pressure. Some ticket categories changed definition in year three when the CSIRT adopted a new taxonomy. OpenTriage-S is a public benchmark with labels created by its authors.

Threats identified:

• Label inaccuracy in tickets (analyst dispositions; category redefinition).
• Dataset shift between NMS-CSIRT and OpenTriage-S (different organization, taxonomy and alert sources); the model is trained on one distribution and tested on another, which is legitimate only if the claim is about transfer.
• Baseline reproduction: the candidate has not rerun the published baseline. If the benchmark's evaluation script or split differs from the one used in the paper, the comparison is invalid. Chapter 4 develops this.
• Base rate: OpenTriage-S is balanced across categories; NMS-CSIRT's real distribution is highly skewed.

Step 3 — Ethics and authorization. The candidate's employment at NMS-CSIRT gives operational access, not research authorization. Required: data-owner approval through a data-use agreement; legal review of using citizen data for model training; ethics review; minimization (remove attachments, replace names with stable pseudonyms, generalize hostnames); a re-identification assessment considering that a municipal CSIRT is small and its incidents may have been reported locally; and an agreement on how trained model weights may be released, since models can memorize training records.

Step 4 — Reproducibility plan.

Lesson table; scroll horizontally to read all columns →

Step 5 — Revised research question. "Does a triage-assistant model trained on one organization's tickets transfer to a different organization's benchmark well enough to exceed a reproduced baseline, and how much does performance change under a time-respecting split and at the deployment base rate?" This is narrower, testable and honest about its limits — and it leaves the bigger question (effect on real response decisions) for a later study design such as the one in Chapter 5.

1.9 Writing a defensible research protocol

A protocol is a public commitment that allows others to audit what was planned against what was done. For an incident-response study, the following structure covers what reviewers, ethics boards and data owners will ask.

Lesson table; scroll horizontally to read all columns →

Two habits distinguish strong protocols. First, they contain predeclared failure conditions: "If inter-rater κ on the pilot is below the target we set in Section 8, we will revise the rubric and repeat the pilot; we will not proceed to the main study." Second, they make the counterfactual explicit: what would the comparison condition look like, and why is it a fair alternative rather than a straw man?

1.10 Practical activity: an offline measurement lab

All records are synthetic. The following Python script uses only the standard library. It computes, for twelve synthetic incidents, (a) a naive mean time-to-contain that drops incidents still open when observation closed, (b) a lower-bound mean that counts open incidents at their observed duration, (c) a Kaplan–Meier median that treats open incidents as censored, (d) Cohen's kappa for two raters who judged each response "adequate" or "inadequate," and (e) positive predictive values at two base rates. Run it offline; it reads and writes no files.

"""OpenCyber Academy doctoral IR, Chapter 1 offline lab. All records are synthetic."""
from collections import Counter

# case_id, hours_to_contain (None = not contained when observation closed), observed_hours, rater_a, rater_b
CASES = [
    ("S01", 2.0, 2.0, "adequate", "adequate"),
    ("S02", 5.5, 5.5, "adequate", "inadequate"),
    ("S03", None, 72.0, "inadequate", "inadequate"),
    ("S04", 1.0, 1.0, "adequate", "adequate"),
    ("S05", 30.0, 30.0, "inadequate", "inadequate"),
    ("S06", None, 48.0, "inadequate", "adequate"),
    ("S07", 8.0, 8.0, "adequate", "adequate"),
    ("S08", 3.0, 3.0, "adequate", "adequate"),
    ("S09", None, 96.0, "inadequate", "inadequate"),
    ("S10", 12.0, 12.0, "adequate", "inadequate"),
    ("S11", 0.5, 0.5, "adequate", "adequate"),
    ("S12", 20.0, 20.0, "inadequate", "inadequate"),
]

def naive_mean_contained(cases):
    xs = [c[1] for c in cases if c[1] is not None]
    return sum(xs) / len(xs), len(xs)

def km_median(cases):
    """Kaplan-Meier product-limit estimate; returns first time S(t) <= 0.5, or None."""
    events = sorted((c[2], c[1] is not None) for c in cases)
    at_risk, surv = len(events), 1.0
    i = 0
    while i < len(events):
        t = events[i][0]
        d = sum(1 for e in events if e[0] == t and e[1])
        n_t = sum(1 for e in events if e[0] == t)
        if d:
            surv *= (at_risk - d) / at_risk
            if surv <= 0.5:
                return t, round(surv, 4)
        at_risk -= n_t
        i += n_t
    return None, round(surv, 4)

def cohen_kappa(a, b):
    n = len(a)
    po = sum(x == y for x, y in zip(a, b)) / n
    ca, cb = Counter(a), Counter(b)
    pe = sum((ca[k] / n) * (cb[k] / n) for k in set(a) | set(b))
    return po, pe, (po - pe) / (1 - pe)

def ppv(sens, fpr, prevalence):
    tp = sens * prevalence
    fp = fpr * (1 - prevalence)
    return tp / (tp + fp)

if __name__ == "__main__":
    mean, k = naive_mean_contained(CASES)
    print(f"contained cases: {k} of {len(CASES)}")
    print(f"naive mean hours (contained only): {mean:.2f}")
    lb = sum(c[2] for c in CASES) / len(CASES)
    print(f"lower-bound mean hours (censored counted at observed time): {lb:.2f}")
    t, s = km_median(CASES)
    print(f"Kaplan-Meier median hours: {t} (S={s})")
    po, pe, kappa = cohen_kappa([c[3] for c in CASES], [c[4] for c in CASES])
    print(f"observed agreement: {po:.3f}; chance agreement: {pe:.3f}; kappa: {kappa:.3f}")
    print(f"PPV at 0.1% prevalence, 99% sensitivity, 1% FPR: {ppv(0.99, 0.01, 0.001):.3f}")
    print(f"PPV at 10% prevalence, 99% sensitivity, 1% FPR: {ppv(0.99, 0.01, 0.10):.3f}")

Expected output (verified by running the script with Python 3):

contained cases: 9 of 12
naive mean hours (contained only): 9.11
lower-bound mean hours (censored counted at observed time): 24.83
Kaplan-Meier median hours: 8.0 (S=0.5)
observed agreement: 0.750; chance agreement: 0.500; kappa: 0.500
PPV at 0.1% prevalence, 99% sensitivity, 1% FPR: 0.090
PPV at 10% prevalence, 99% sensitivity, 1% FPR: 0.917

Interpretation. The naive mean (9.11 hours) silently removes the three slowest incidents. Counting them at their observed durations raises the mean to 24.83 hours, and even that is only a lower bound, because those incidents were still open. The true mean cannot be estimated from these data without further follow-up or a model for the tail. The Kaplan–Meier median (8.0 hours) uses the censored cases correctly as "at risk" until they leave observation and is the most defensible single summary here. The raters agreed on 9 of 12 cases (75%), but because both rated roughly half the cases "adequate," half of that agreement is expected by chance, giving κ = 0.50: the rubric needs work before it can support an outcome analysis. Finally, the same classifier performance yields a very different value to responders depending on prevalence.

1.11 Using the synthetic case corpus as a methods resource

The lower-level books in this series built three synthetic cases: Harbor Library (rapid), Cedar Training Network (professional) and Lumen Library Reservations (technical). At doctoral level they are useful as scenario seeds for simulated study conditions — for example, as tabletop cases presented to responders under two workflows in Chapter 5. They are not data. Their timelines were written to teach, so their ground truth is known and their difficulty was chosen by an author. Any measurement taken on them is evidence about how participants perform on those scenarios, under those conditions. A thesis that uses them should cite them as instruments, document which version was used, and resist the temptation to generalize from "responders did better on the Harbor scenario" to "responders do better."

Common errors and limits

• Treating operational access to incident records as research authorization.
• Calling data "anonymized" because names were removed, without a re-identification assessment that considers organizations as well as individuals.
• Reporting a time metric without defining its start and end events or handling censored incidents.
• Using analyst dispositions as ground truth without auditing them.
• Writing a preregistration after exploring the confirmation data, or omitting the deviation log.
• Citing a vendor caseload report as a population estimate.
• Coding a single "root cause" per incident without justifying the modeling choice.
• Generalizing from synthetic or exercise scenarios to field performance.
• Attaching a citation that has not been read, or one generated by a tool and not verified.
• Limit of this chapter: it gives structures for legal and ethical analysis but is not legal advice and does not replace an institutional review.

Exercises — chapter 1

EX001 — Claim ladder audit. Objective: separate observation, inference and claim in incident writing. Inputs: the synthetic post-incident paragraph below. Constraints: use only the paragraph; do not add facts. Steps: (1) split the paragraph into atomic statements; (2) label each as report, observation, inference, case claim or research claim (§1.2.2); (3) for each inference, name its hidden assumption and one defeating observation; (4) rewrite the final sentence as a scoped claim. Expected evidence: a table of statements with labels, assumptions and defeaters, plus the rewritten claim. Rubric: 3 = every statement correctly labeled, every inference has a testable defeater, rewrite is scoped; 2 = one mislabel or vague defeater; 1 = labels present but inferences treated as observations; 0 = no separation.

Synthetic paragraph: "At 02:14 the EDR console showed powershell.exe launched by winword.exe on host FIN-22. The user had opened a phishing email. The attacker then moved laterally to FIN-31 using stolen credentials, as shown by a 02:40 logon event. Our new email filter, deployed last month, would have blocked the message. This proves that email filtering reduces ransomware risk."

EX002 — Hindsight-resistant interview guide. Objective: design data collection that limits outcome bias. Inputs: a synthetic incident timeline with five time-stamped artifacts of your choosing (alert, ticket note, chat message, containment action, status update). Constraints: the guide must be usable 2–4 weeks after the incident; no question may reveal the final outcome before asking about earlier reasoning. Steps: (1) write six to eight questions anchored to the artifacts in time order; (2) mark which questions risk eliciting post hoc rationalization and how you mitigate it; (3) write a one-paragraph consent statement that addresses blame and employment risk. Expected evidence: the guide, mitigation notes and consent paragraph. Rubric: 3 = time-anchored, outcome-blind ordering, explicit mitigation, consent addresses blame and withdrawal; 2 = one of these weak; 1 = generic questions; 0 = leading questions or outcome revealed first.

EX003 — Constraint mapping and tiered release. Objective: produce an honest open-science plan for restricted incident data. Inputs: the NMS-CSIRT worked example in §1.8, plus the following added facts: two incidents are under a litigation hold; one threat-intelligence feed used in tickets was received as TLP:AMBER; the CSIRT's MSSP contract forbids naming the MSSP. Constraints: use the constraint taxonomy (§1.4.3) and tiers (§1.4.5). Steps: (1) list each artifact the thesis would produce; (2) map each to its binding constraints; (3) assign a release tier and justify it; (4) draft the thesis's "Open Science" paragraph (150–250 words). Expected evidence: artifact–constraint–tier table and paragraph. Rubric: 3 = every constraint correctly applied, tiers maximal but lawful, paragraph states what reviewers cannot check; 2 = one misapplied constraint; 1 = blanket "data unavailable"; 0 = proposes releasing restricted material.

EX004 — Ethical Impact Assessment. Objective: apply the four Menlo principles to a concrete study. Inputs: a proposed study in which a researcher will observe NMS-CSIRT's on-call responders during live incidents for three months and record decision points. Constraints: use the Menlo structure (§1.5.2) and authorization layers (§1.5.3); the response must always take priority. Steps: (1) list stakeholders and their risks; (2) for each principle, state the main risk and a safeguard; (3) list required authorizations in order; (4) write five stop rules; (5) state the incidental-discovery procedure. Expected evidence: a two-page assessment. Rubric: 3 = stakeholders include data subjects, responders, organization and public; safeguards specific; stop rules testable; 2 = one stakeholder group or layer missing; 1 = generic ethics text; 0 = treats manager approval as sufficient consent.

EX005 — Measurement lab. Objective: quantify how censoring, rater agreement and base rates change conclusions. Inputs: the script in §1.10. Constraints: run offline; record exact outputs. Steps: (1) run the script and confirm the expected output; (2) in a copy, delete the three censored cases entirely and recompute the Kaplan–Meier median; (3) in another copy, change rater B's judgment of S10 to "adequate" and recompute kappa; (4) compute PPV for 99% sensitivity and 1% FPR at 1% prevalence; (5) write 150 words interpreting all three changes. Expected evidence: the outputs and interpretation. Rubric: 3 = exact values and correct interpretation of bias direction; 2 = values correct, interpretation thin; 1 = one value wrong; 0 = no runs.

EX006 — Failure-mode review of a claim. Objective: apply the failure-mode catalog (§1.7) as a reviewer. Inputs: this synthetic abstract: "We surveyed 212 security leaders. Organizations that ran quarterly tabletop exercises reported 40% faster containment than those that did not. Tabletop exercises therefore cut containment time by 40%." Constraints: identify at least five distinct failure modes; do not invent additional data. Steps: (1) name each failure mode and quote the phrase that triggers it; (2) explain the alternative explanation it opens; (3) propose a design change; (4) rewrite the conclusion to match the evidence. Expected evidence: review table and rewritten conclusion. Rubric: 3 = five or more correctly identified with design changes and a properly scoped rewrite; 2 = three or four; 1 = generic criticism; 0 = accepts the claim.

EX007 — Defensible protocol skeleton. Objective: write the first version of a protocol for the revised NMS-CSIRT question (§1.8, Step 5). Inputs: the sixteen-section template in §1.9. Constraints: 1,200–2,000 words; at least three predeclared failure conditions; the analysis plan must handle censoring if any time measure is used and must state base rates. Steps: complete every section; mark assumptions you could not verify. Expected evidence: versioned protocol (v0.1) with a deviation-log template. Rubric: 3 = all sections, claim matches design, failure conditions testable, release tiers and stop rules present; 2 = one or two sections thin; 1 = several missing; 0 = protocol claims effect on real response from benchmark accuracy.

Chapter synthesis

Incident-response research studies events that happen once, are shaped by the response itself, are read with hindsight and are wrapped in legal and ethical restrictions. Those conditions do not excuse weak claims; they demand explicit ones. The researcher's basic tools are the observation–inference–claim distinction, pre-specification with a transparent deviation log, a tiered reproducibility plan that releases what can safely be released, layered authorization and an Ethical Impact Assessment, measures whose validity is argued rather than assumed, and a protocol that states in advance what would count as failure. Chapter 2 uses these tools to read the research literature and locate a question that is genuinely open.