OpenCyber Academy
Postgraduate · Chapter 10 · English
Course code: OCA-153Chapter 10: OCA-153-C010

Chapter 10 — Research reading and critique

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 10 published. Map, example, EX046.

flowchart TD
  N0["Learning outcomes"]
  N1["10.1 Why read research"]
  N2["10.2 Paper one"]
  N3["10.3 Paper two"]
  N4["10.4 Paper three"]
  N5["10.5 Critique structure"]
  N6["10.6 Relating to Meridian"]
  N7["10.7 Avoiding cherry-picking"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX046"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["10.1 Why read research"]
  N2["10.2 Paper one"]
  N3["10.3 Paper two"]
  N4["10.4 Paper three"]
  N5["10.5 Critique structure"]
  N6["10.6 Relating to Meridian"]
  N7["10.7 Avoiding cherry-picking"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX046"]

[Learning outcomes]
    |
    v
[10.1 Why read research]
    |
    v
[10.2 Paper one]
    |
    v
[10.3 Paper two]
    |
    v
[10.4 Paper three]
    |
    v
[10.5 Critique structure]
    |
    v
[10.6 Relating to Meridian]
    |
    v
[10.7 Avoiding cherry-picking]
    |
    v
[Worked example] --> [Exercise EX046]

Learning outcomes

You will (1) read SOC research papers critically; (2) assess methods, samples and claims; (3) relate findings to one's own SOC; (4) write a structured critique.

10.1 Why read research

Research can test ideas SOCs adopt on faith. Daniel's programme asks students to critique three papers relevant to their work. For Meridian, research matters when it challenges a practice the team takes for granted, such as counting every benign alert as a false positive. Daniel chose papers on analyst experience, false positives and alert ranking because each touches a decision his SOC makes every week. His critiques end with a concrete action or a reason for taking none, so the reading changes practice rather than only knowledge.

10.2 Paper one

Kokulu et al. (2019) interviewed SOC staff and found mismatches between managers and analysts on issues such as metrics and automation. Strength: rich qualitative data from several SOCs. Limit: interview samples may not represent all SOCs. For Meridian, the paper's relevance is the mismatch on metrics: Daniel's managers value acknowledgement times, while his analysts may experience them as pressure that ignores difficult cases. His planned survey of his own analysts tests whether the mismatch exists at Meridian before changing any metric. A finding from other SOCs is a reason to ask, not an answer.

10.3 Paper two

Alahmadi, Axon and Martinovic (2022) surveyed and interviewed SOC practitioners about false positives, finding many are benign triggers. Strength: distinguishes types of false positive. Limit: self-reported perceptions, not measured alert data. The distinction matched Meridian's experience directly: h04's flag was a benign trigger, a correct detection of activity that had a business reason. Daniel's critique concludes that Meridian will record benign triggers separately from false positives in closure codes. Because the paper measured perceptions, he will check his own closure data before changing any rule.

10.4 Paper three

Hassan et al. (2019) proposed NoDoze, ranking alerts using provenance graphs, and evaluated it on enterprise data. Strength: quantitative evaluation. Limit: requires detailed provenance data many SOCs do not collect. For Meridian, the method's provenance requirement is the obstacle: the SOC does not collect detailed process provenance across endpoints, so NoDoze could not be applied without new data collection. Daniel records the idea of ranking alerts by context as worth watching, and notes that Meridian's assignment enrichment is a much simpler step in the same direction. His critique states that a quantitative evaluation elsewhere does not transfer automatically to a different data set.

10.5 Critique structure

Question, method, sample, findings, validity, relevance to practice and open questions. Daniel applies the structure to each paper in the same order, which makes his three critiques comparable and shows where each is strong or weak. For Alahmadi et al., the validity point was that results are perceptions; for NoDoze, relevance to practice was limited by data Meridian lacks. The open questions in each critique became items he could test with Meridian's own data. Using one structure also makes it easy to see when a critique skips a step, such as validity.

10.6 Relating to Meridian

Kokulu's mismatch finding prompts Daniel to survey his own analysts about metrics; Alahmadi's distinction informs how Meridian labels closed alerts. Neither change is adopted on the papers' authority alone. The survey checks whether Kokulu's finding holds at Meridian, and the closure-label change will be followed by a look at Meridian's own closure data before rules are modified. Research suggests where to look; local evidence decides what to do. Daniel records both steps in his critique, so Prof. Achebe can see how each paper led to a specific action.

10.7 Avoiding cherry-picking

Daniel searches for studies disagreeing with each paper's conclusions and notes them. For each paper he records at least one study that found differently or questioned the method, and states whether it changes his conclusion. This guards against choosing papers that confirm what he already believed, which is easy when the reading list is chosen by the reader. Prof. Achebe asks students to submit the disagreeing studies with their critiques. Where no disagreeing study could be found, he says so in the critique, rather than implying the conclusion is settled.

Reading groups

Daniel starts a monthly reading group where analysts present one paper and its relevance. Reading together spreads critical habits beyond the leader and often surfaces practical objections that a single reader would miss, such as data the SOC does not actually collect.

Worked example

Daniel's critique of the Alahmadi paper concludes: "The benign-trigger distinction is directly useful; we will add 'benign trigger' as a closure code separate from 'false positive'. Because results are perceptions, we will measure our own closure data before changing rules."

Synthetic case. Meridian's actions are invented; the paper summaries describe the cited works.

Errors and limits

Critiques by practitioners may overweight relevance and underweight rigour; balance both.

Exercises

Synthesis

Critical reading tests ideas before adoption and links research findings to measured local practice.

Continue studying — chapter 10 of 12

← Chapter 09 (draft) · Course chapter index · Chapter 11 (draft) →

Worked answers

EX046. SOC managers and analysts disagree on issues like metrics and automation; interview samples may not represent all SOCs.

EX047. It relies on practitioners' perceptions rather than measured alert data.

EX048. It depends on detailed provenance data many SOCs do not collect.

EX049. Question, method, sample, findings, validity, relevance to practice and open questions.

EX050. Add a separate "benign trigger" closure code and measure Meridian's own closure data before changing rules.