Chapter 10 — Research reading and critique

Chapter 10 — Research reading and critique

Lesson map

Lesson 10 published. Map, example, EX046.

flowchart TD
  N0["Learning outcomes"]
  N1["10.1 Reading with purpose"]
  N2["10.2 Judging transfer"]
  N3["10.3 Writing the critique"]
  N4["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4
  N4 --> EX["Worked example"]
  EX --> Q["Exercise EX046"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["10.1 Reading with purpose"]
  N2["10.2 Judging transfer"]
  N3["10.3 Writing the critique"]
  N4["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX046"]

[Learning outcomes]
    |
    v
[10.1 Reading with purpose]
    |
    v
[10.2 Judging transfer]
    |
    v
[10.3 Writing the critique]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX046]

Learning outcomes

You will (1) read research papers on security architecture critically; (2) judge whether findings transfer to Corvane; (3) write a structured critique.

10.1 Reading with purpose

Dr Voss sets a reading task: find and critique published research on whether segmentation or zero trust designs reduce incident impact. Priya reads with three questions. What exactly was measured? How was the comparison made? Would the setting resemble a clinic network? She notes that many papers measure simulated networks or vendor customer bases, and few measure health care. Herley and van Oorschot (2017) argue that security research often lacks falsifiable claims, so she looks for studies that state what result would have contradicted them.

10.2 Judging transfer

A study of large enterprises finds that segmented networks contain lateral movement faster. Priya asks whether this transfers to Corvane: her clinics are smaller, have legacy devices and rely on a central electronic record. Some mechanisms transfer, such as limiting reachable hosts; others do not, such as dedicated security operations staff watching every alert. She concludes that the direction of the finding likely applies but the size of the effect is uncertain. Transfer judgements are reasoned arguments about mechanisms, not simple yes or no answers.

10.3 Writing the critique

Priya writes a two-page critique for each paper: claim, method, data, threats to validity, transfer to Corvane and what further evidence would help. She avoids dismissing work for imperfections every study has and instead judges whether flaws undermine the claim. One paper's conclusions rest on an uncontrolled before-and-after comparison during a period of other changes, which she marks as weak. Another uses a careful comparison group, which she marks as stronger despite a smaller sample. Critique is about weighing evidence, not scoring points.

Worked example

Priya critiques research on segmentation with three reading questions, reasons about transfer by mechanism and writes structured critiques.

Synthetic case. The papers described are synthetic composites; Herley and van Oorschot (2017) is real.

Errors and limits

Critiques can be harsher on unwelcome findings. Transfer reasoning is itself uncertain.

Exercises

• EX046 — Three questions. What three questions guide Priya's reading? Rubric: 3 = what was measured, how compared, would the setting resemble a clinic;; 2 = two;; 1 = vague;; 0 = none.
• EX047 — Falsifiable claims. Why look for studies stating what would contradict them? Rubric: 3 = security research often lacks falsifiable claims;; 2 = falsifiable;; 1 = vague;; 0 = none.
• EX048 — Mechanism not transferring. Which mechanism from the enterprise study did not transfer? Rubric: 3 = dedicated security operations staff watching alerts;; 2 = staffing;; 1 = segmentation;; 0 = none.
• EX049 — Transfer conclusion. What did Priya conclude about transfer? Rubric: 3 = direction likely applies, effect size uncertain;; 2 = direction;; 1 = full transfer;; 0 = none.
• EX050 — Weak design. Why was one paper marked weak? Rubric: 3 = uncontrolled before-and-after comparison during other changes;; 2 = uncontrolled;; 1 = small sample;; 0 = none.

Synthesis

Critical reading asks what was measured, how and whether mechanisms transfer, and weighs flaws by their effect on claims.

Continue studying — chapter 10 of 12

← Chapter 09 (draft) · Course chapter index · Chapter 11 (draft) →