OpenCyber Academy
Postgraduate · Chapter 10 · English
Course code: OCA-156Chapter 10: OCA-156-C010

Chapter 10 — Research reading and critique

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 10 published. Map, example, EX046.

flowchart TD
  N0["Learning outcomes"]
  N1["10.1 Types of source"]
  N2["10.2 Method questions"]
  N3["10.3 Transferability"]
  N4["10.4 Reading foundational models critically"]
  N5["10.5 Structured critique"]
  N6["10.6 Building a reading habit"]
  N7["Finding good sources"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX046"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["10.1 Types of source"]
  N2["10.2 Method questions"]
  N3["10.3 Transferability"]
  N4["10.4 Reading foundational models critically"]
  N5["10.5 Structured critique"]
  N6["10.6 Building a reading habit"]
  N7["Finding good sources"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX046"]

[Learning outcomes]
    |
    v
[10.1 Types of source]
    |
    v
[10.2 Method questions]
    |
    v
[10.3 Transferability]
    |
    v
[10.4 Reading foundational models critically]
    |
    v
[10.5 Structured critique]
    |
    v
[10.6 Building a reading habit]
    |
    v
[Finding good sources]
    |
    v
[Worked example] --> [Exercise EX046]

Learning outcomes

You will (1) read academic and practitioner CTI research critically; (2) evaluate methods and data; (3) assess transferability to your organisation; (4) write a structured critique.

10.1 Types of source

Peer-reviewed papers, standards (STIX, TAXII, SP 800-150), government guidance (ICD 203), foundational practitioner papers (Hutchins et al.; Caltagirone et al.) and vendor reports differ in review, incentives and evidence. Know which you are reading. Rafael's team reads each type of source differently, with its incentives in mind. Peer-reviewed papers explain their method; standards define formats but not effectiveness; government guidance sets analytic standards; practitioner papers give models; vendor reports may be shaped by commercial interest. When the team read a vendor study claiming its feed detects 90% of threats, they checked its method before trusting the claim.

10.2 Method questions

What data was used, from where, over what period? How were labels created? What comparison was made? Are results reproducible? Could the authors' interests shape the conclusion? Applied to the vendor study, these questions showed that threats were defined as items in the vendor's own catalogue, so the 90% figure measured the feed against itself. The team could not reproduce the result outside the vendor. Rafael records the answers to each question when the team reviews a source, so later readers can see why it was trusted or not.

10.3 Transferability

A study on a large technology company's feeds may not transfer to a mid-sized logistics firm. Ask whether populations, threat exposure and resources are similar. Before applying any study's results at NLG, Rafael's team asks whether the studied organisations face similar threats, use similar tools and have similar staff. A finding about a large technology company's feed may not hold for a logistics firm with a small CTI team. Where transfer is uncertain, the team tests the idea on NLG's own data first.

10.4 Reading foundational models critically

The kill chain was written for intrusion campaigns against defence contractors; it fits insider or cloud-identity attacks less well. The Diamond Model is a framework for organising events, not a method that produces conclusions by itself. ATT&CK reflects publicly reported behaviour, biased toward what vendors observe. At NLG, the kill chain helps describe a ransomware intrusion but fits poorly for supplier credential misuse. Rafael uses the Diamond Model to organise events from the data theft, but conclusions still come from ACH. ATT&CK frequencies in his project are labelled as reflecting reported behaviour, not all activity.

10.5 Structured critique

Summary, contribution, method strengths, method weaknesses, transferability, and what you would do with it. Keep tone professional. Rafael's team writes a one-page critique of each paper read in the monthly group, following this structure. The vendor study critique concluded that its claim was unverifiable externally, and the team decided not to rely on it in procurement. Critiques are stored with the source catalogue, so future readers find the team's view next to the source.

Critiques avoid personal comments about authors and focus on whether the method supports the claims.

10.6 Building a reading habit

A monthly reading group of two papers keeps the team current. Each session ends with one change to try or a reason not to. At NLG, the group meets monthly and picks papers related to current requirements, such as technique frequency or triage evaluation. Ending each session with one change to try links reading to practice. Rafael records the changes tried and whether they helped, so the group can see which papers influenced NLG's work.

Analysts take turns choosing papers, so the reading reflects different parts of the team's work.

Finding good sources

Use scholarly databases and conference proceedings for peer-reviewed work, standards bodies for specifications and official guidance, and reputable practitioner venues for applied studies. Record why each source was selected, and prefer primary sources over summaries that may distort them.

Worked example

Rafael's team reads a vendor study claiming its feed "detects 90% of threats". The method section defines threats as items in the vendor's own catalogue — a circular measure. The team notes the claim is unverifiable externally and decides not to rely on it in procurement.

Synthetic case. The vendor study is invented.

Errors and limits

Critique is easier than building; pair every criticism with a constructive suggestion.

Exercises

Synthesis

Read sources by type, interrogate methods, check transferability and critique constructively.

Continue studying — chapter 10 of 12

← Chapter 09 (draft) · Course chapter index · Chapter 11 (draft) →

Worked answers

EX046. The study counts as "threats" only what the vendor already catalogues, so detection is measured against its own definition.

EX047. What data, from which sources and period; how labels were made; what comparison was used; whether it can be reproduced; and whether authors' interests could shape findings.

EX048. Insider misuse and cloud-identity attacks, which skip phases such as delivery and exploitation.

EX049. Whether the studied organisations resemble NLG in size, sector threat exposure and security resources.

EX050. No. It organises events into adversary, capability, infrastructure and victim; conclusions come from the analyst's reasoning over that structure.