Chapter 37 — Evidence review seminar

Chapter 37 — Evidence review seminar

Lesson map

Lesson 37 published. Map, example, EX109.

flowchart TD
  N0["Learning outcomes"]
  N1["37.1 Appraising sources"]
  N2["37.2 Types of source"]
  N3["37.3 Presenting fairly"]
  N4["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4
  N4 --> EX["Worked example"]
  EX --> Q["Exercise EX109"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["37.1 Appraising sources"]
  N2["37.2 Types of source"]
  N3["37.3 Presenting fairly"]
  N4["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX109"]

[Learning outcomes]
    |
    v
[37.1 Appraising sources]
    |
    v
[37.2 Types of source]
    |
    v
[37.3 Presenting fairly]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX109]

Learning outcomes

You will (1) appraise published sources critically; (2) distinguish standards, research and opinion; (3) present a fair evidence summary to peers.

37.1 Appraising sources

In the seminar, each student reviews evidence on a question. Lucas takes the question: does network segmentation reduce the spread of ransomware in small organisations? He collects standards, vendor reports, academic papers and blog posts. For each he asks who wrote it, what method was used, what data supports it and what interests the author has. Herley and van Oorschot (2017) warn that much security knowledge rests on assertion rather than tested claims, so Lucas looks for evidence that would survive scrutiny.

37.2 Types of source

Lucas sorts his sources. Standards such as SP 800-53 Rev. 5 and ISO/IEC 27002:2022 record expert consensus about good practice but do not measure outcomes. Vendor reports offer large data sets but have commercial interests and unclear methods. Academic studies have clearer methods but often study simulated networks. Blog posts give vivid examples but cannot be generalised. He weighs each type for what it can show. The result is a picture in which segmentation is widely recommended and plausible, but outcome evidence for small organisations is thin.

37.3 Presenting fairly

Lucas presents his summary to the seminar with a table showing each source, its type, its finding and its limits. He states his conclusion carefully: segmentation is reasonable practice supported by expert consensus and simulation studies, but real-world measurements for small organisations are scarce. A classmate challenges him with a vendor statistic; he explains why its method is unclear. Dr Bellini praises the fairness, noting that an honest summary of weak evidence is more useful to practitioners than a confident one built on marketing.

Worked example

Lucas appraises sources on segmentation and ransomware, classifies them by type and presents a fair summary noting thin outcome evidence.

Synthetic case. The source collection is synthetic in detail; Herley and van Oorschot (2017), SP 800-53 Rev. 5 and ISO/IEC 27002:2022 are real.

Errors and limits

Absence of evidence is not evidence of absence. Reviewers can carry their own bias.

Exercises

• EX109 — Four questions. What four questions did Lucas ask of each source? Rubric: 3 = author, method, data and interests;; 2 = three;; 1 = vague;; 0 = none.
• EX110 — Standards limit. What do standards not do? Rubric: 3 = measure outcomes;; 2 = outcomes;; 1 = everything;; 0 = none.
• EX111 — Conclusion. What was Lucas's conclusion on segmentation? Rubric: 3 = reasonable practice backed by consensus and simulations, but thin real-world evidence for small organisations;; 2 = balanced;; 1 = proven;; 0 = none.

Synthesis

Fair evidence review weighs each source type for what it can show and states weak evidence honestly.

Continue studying — chapter 37 of 40

← Chapter 36 (draft) · Course chapter index · Chapter 38 (draft) →