Chapter 1 — Advanced research integrity and methods
Lesson map
Lesson 01 published. Map, example, EX001.
flowchart TD
N0["Learning outcomes"]
N1["1.1 Why integrity matters in SOC research"]
N2["1.2 Three constructs, kept apart"]
N3["1.3 Data for research"]
N4["1.4 Pre-registration"]
N5["1.5 Interests and tools"]
N6["1.6 Ethics with analysts as participants"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX001"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["1.1 Why integrity matters in SOC research"]
N2["1.2 Three constructs, kept apart"]
N3["1.3 Data for research"]
N4["1.4 Pre-registration"]
N5["1.5 Interests and tools"]
N6["1.6 Ethics with analysts as participants"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX001"]
[Learning outcomes]
|
v
[1.1 Why integrity matters in SOC research]
|
v
[1.2 Three constructs, kept apart]
|
v
[1.3 Data for research]
|
v
[1.4 Pre-registration]
|
v
[1.5 Interests and tools]
|
v
[1.6 Ethics with analysts as participants]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX001]
Learning outcomes
You will (1) apply research-integrity principles to security-operations research; (2) separate detection quality, triage quality and operational outcome as distinct constructs; (3) design studies that use synthetic or consented operational data; (4) pre-register hypotheses, measures and analysis.
1.1 Why integrity matters in SOC research
SOC research shapes how organisations spend scarce analyst time. A study that overstates a triage aid can lead a SOC to switch off useful alerts or trust a tool it should check. Vielberth et al. (2020), in a systematic study of SOC literature, found the field fragmented and short of empirical evaluation; Kokulu et al. (2019) showed through interviews that managers and analysts often disagree about what the SOC's problems even are. A doctoral project should add careful evidence, not more opinion.
1.2 Three constructs, kept apart
Detection quality asks whether a rule fires on the activity it targets and stays quiet otherwise (precision, recall). Triage quality asks whether an analyst, given an alert, reaches the right disposition (accuracy, time, agreement). Operational outcome asks whether incidents are contained sooner or with less harm. A study can improve one while harming another; claims must name which construct was measured. Nadia's thesis measures triage quality only. Her synthetic alerts have known labels, so accuracy can be scored exactly, but nothing in the design shows whether Northgate would contain incidents faster. Her conclusion chapter keeps that boundary.
1.3 Data for research
Real SOC data contains personal data, business secrets and evidence of live incidents. It is rarely shareable and may not be consented for research. Synthetic alert sets with known ground truth allow exact scoring and open publication. Where operational data is needed, it is used under a data-sharing agreement, minimised and pseudonymised, with ethics approval. Nadia's corpus is synthetic for these reasons: Northgate serves several municipalities, and its real alerts would include residents' and staff data. The only operational data in her study are the analysts' own triage logs, collected with consent and anonymised before analysis, under ethics approval from Calder University.
1.4 Pre-registration
Before the main study the candidate registers hypotheses, primary and secondary measures, sample size reasoning, analysis plan and exclusion rules. Later changes are allowed but logged with reasons, so readers can tell planned from exploratory results. Nadia registers accuracy as the primary measure and time-to-disposition as secondary, with the paired analysis and the exclusion rule for incomplete sessions. Her change log, starting with version 1.1 after the pilot, records each later change, so readers can see that moving context fields to the top of the card happened before the main study, not after the results were known.
1.5 Interests and tools
Vendors may fund SOC studies or supply tools. Funding, relationships and tool versions are declared. A study that tests a product must not let the vendor approve conclusions. Nadia's study uses no vendor product; the alert cards are her own design, and the rules are written in the open Sigma format. She still records the Python version used to generate alerts and score decisions, and Calder's declaration form states that Northgate receives no payment or product benefit from the study. Her thesis lists these declarations in the front matter, so examiners see them before the results.
1.6 Ethics with analysts as participants
When analysts are participants, their performance data could affect careers. Consent is voluntary, managers do not see individual scores, and participants may withdraw. At Northgate, Ruth Okonkwo approved participation in work time but agreed in writing not to see individual accuracy scores. Analysts receive random IDs before the sessions, and results are reported only in aggregate. An analyst who withdraws can have their logs removed up to the point of analysis, and the consent form states that date. Nadia also promised to present results to the analysts first.
Worked example
Nadia Ferreira, a doctoral candidate at the fictional Calder University supervised by Prof. Elias Grant, drafts the aim "context-enriched alerts make analysts better". Grant asks: better at what, measured how, against what truth, with what uncertainty? Nadia rewrites: "Estimate the difference in triage accuracy and median time-to-disposition between baseline and context-enriched alert cards on 160 synthetic alerts with known labels, reporting 95% intervals."
Synthetic case. Calder University, Nadia, Prof. Grant and all data are invented.
Errors and limits
Synthetic alerts simplify real operations. Results transfer only as far as the alert set resembles a real SOC's mix; the thesis must say which features were not modelled.
Exercises
- EX001 — Three constructs. Distinguish detection quality, triage quality and operational outcome. Rubric: 3 = all three with a measure each; 2 = two; 1 = one; 0 = none.
- EX002 — The field's gap. What did Vielberth et al. (2020) report about SOC research? Rubric: 3 = fragmented, short of empirical evaluation; 2 = partial; 1 = vague; 0 = none.
- EX003 — Perspectives differ. What did Kokulu et al. (2019) find about managers and analysts? Rubric: 3 = they often disagree on the SOC's problems; 2 = partial; 1 = vague; 0 = none.
- EX004 — Rewrite the aim. Turn "enriched alerts make analysts better" into a testable aim. Rubric: 3 = measures, conditions, ground truth, uncertainty; 2 = three; 1 = vague; 0 = none.
- EX005 — Pre-registration content. Name five things registered before the main study. Rubric: 3 = hypotheses, measures, sample-size reasoning, analysis, exclusions; 2 = three; 1 = one; 0 = none.
- EX006 — Participant protection. How are analyst participants protected? Rubric: 3 = voluntary consent, no individual scores to managers, withdrawal; 2 = two; 1 = one; 0 = none.
- EX007 — Synthetic boundary. State the main limit of synthetic alert sets. Rubric: 3 = transfer only as far as the mix resembles reality; say what is missing; 2 = partial; 1 = vague; 0 = none.
Synthesis
SOC research must name the construct measured, use shareable or properly governed data, pre-register and protect participants.