Chapter 37 — Evidence review seminar
Lesson map
Lesson 37 published. Map, example, EX109.
flowchart TD
N0["Learning outcomes"]
N1["37.1 Critiquing published evidence"]
N2["37.2 Comparing studies fairly"]
N3["37.3 Applying evidence to decisions"]
N4["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4
N4 --> EX["Worked example"]
EX --> Q["Exercise EX109"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["37.1 Critiquing published evidence"]
N2["37.2 Comparing studies fairly"]
N3["37.3 Applying evidence to decisions"]
N4["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX109"]
[Learning outcomes]
|
v
[37.1 Critiquing published evidence]
|
v
[37.2 Comparing studies fairly]
|
v
[37.3 Applying evidence to decisions]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX109]
Learning outcomes
You will (1) critique published evidence; (2) compare studies fairly; (3) apply evidence to practice decisions.
37.1 Critiquing published evidence
In a seminar, Sofia's group critiques Durieux and colleagues' 2020 tool evaluation. They ask: what contracts were used, how were weaknesses confirmed, what counted as a correct detection, and could the results generalise to other contracts or newer tools? They note strengths, such as a large dataset and published materials, and limitations, such as the difficulty of knowing the true set of weaknesses. Critique does not mean dismissing a study; it means understanding exactly what it shows. Each student writes a one-page critique following a common structure provided by Dr Asante.
37.2 Comparing studies fairly
The group compares Durieux and colleagues' findings with Perez and Livshits' 2021 study of whether reported weaknesses were exploited. The studies ask different questions: one about tool accuracy, the other about real-world exploitation. Sofia notes that combining them suggests tools flag many issues, some of which are false or unexploited, while still missing others. Fair comparison respects each study's design and does not force agreement where questions differ. The group writes a short synthesis noting where the evidence agrees, where it differs and where it is silent.
37.3 Applying evidence to decisions
The cooperative asks whether to buy a commercial analysis service. Sofia uses the seminar's synthesis: tools catch some issues but miss others, so any service should complement, not replace, manual review and testing. Given the contract's small size and the cooperative's budget, she recommends one external manual review before major changes rather than a subscription. She cites the studies in her recommendation. Applying evidence to decisions is the purpose of reviewing it, and she records the decision and evidence together for later reference by the board.
Worked example
Sofia's group critiques Durieux et al. and compares it with Perez and Livshits, leading her to recommend one external manual review before major changes.
Synthetic case. The seminar and recommendation are synthetic; Durieux et al. (2020) and Perez and Livshits (2021) are real sources.
Errors and limits
Forcing studies with different questions into agreement misrepresents both.
Exercises
- EX109 — Critique question. Name one question asked of the tool study. Rubric: 3 = how weaknesses were confirmed;; 2 = what contracts were used;; 1 = who funded the seminar;; 0 = none.
- EX110 — Study difference. How do the two studies differ? Rubric: 3 = tool accuracy versus real-world exploitation;; 2 = different questions;; 1 = identical;; 0 = none.
- EX111 — Service recommendation. What does Sofia recommend instead of a subscription? Rubric: 3 = one external manual review before major changes;; 2 = manual review;; 1 = buy the service;; 0 = none.
Synthesis
Fair critique and synthesis of evidence support practical security decisions.
Continue studying — chapter 37 of 40
← Chapter 36 (draft) · Course chapter index · Chapter 38 (draft) →