Chapter 37 — Evidence review seminar
Lesson map
Lesson 37 published. Map, example, EX109.
flowchart TD
N0["Learning outcomes"]
N1["37.1 Choosing studies"]
N2["37.2 Designs"]
N3["37.4 Synthesis"]
N4["37.5 Implications"]
N5["37.6 Presentation"]
N6["Reading critically"]
N7["Selection criteria"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX109"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["37.1 Choosing studies"]
N2["37.2 Designs"]
N3["37.4 Synthesis"]
N4["37.5 Implications"]
N5["37.6 Presentation"]
N6["Reading critically"]
N7["Selection criteria"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX109"]
[Learning outcomes]
|
v
[37.1 Choosing studies]
|
v
[37.2 Designs]
|
v
[37.4 Synthesis]
|
v
[37.5 Implications]
|
v
[37.6 Presentation]
|
v
[Reading critically]
|
v
[Selection criteria]
|
v
[Worked example] --> [Exercise EX109]
Learning outcomes
You will (1) critically review published evidence on supply-chain security; (2) compare study designs and limits; (3) synthesise findings for practice; (4) present a balanced review.
37.1 Choosing studies
Lena's seminar group reviews five studies: Ohm et al. (2020) on malicious packages, Zimmermann et al. (2019) on npm ecosystem risk, Decan et al. (2018) on vulnerability propagation, Kula et al. (2018) on dependency updating and Wermke et al. (2022) on open-source trust.
The group chooses studies that cover different parts of the supply chain and different research designs, so the review is not dominated by one method or one kind of evidence. Each member reads two studies in full and summarises them using a common template covering question, data, method, findings and limits.
37.2 Designs
Two are case collections, two are large-scale measurement studies, one is an interview study. Each design answers different questions: what happens, how often, and why.
Knowing the design tells the reader what a study can and cannot claim. Case collections can show what attacks look like but not how common they are; measurement studies show scale but not motivation; interviews explain reasons but cannot show frequency. Lena's group marks each finding with its design, so readers can judge its strength.
37.3 Limits
Case collections show only discovered cases; measurement studies depend on data quality and ecosystem; interview studies have small, self-selected samples.
The group also notes that most studies examine the npm ecosystem, which may differ from others CampusEats depends on, and that ecosystems and registry policies have changed since some of the data was collected. Limits do not make a study useless, but they set the boundary of what it can support, and the group states each limit next to the finding it affects.
37.4 Synthesis
Together they suggest: risk concentrates in widely used packages; updates lag; malicious packages exploit trust and install-time behaviour; maintainers vary widely in security practice.
Each point is supported by at least two studies of different designs, which makes it stronger than a finding from one source. The group separates these from points supported only once, such as specific maintainer practices, and presents them as less certain. They also note questions none of the studies answered, such as how small teams decide which updates to trust.
37.5 Implications
For small projects: fewer dependencies, prompt updates, install-time restrictions, and checks on new names and maintainers.
Each implication is something CampusEats already does or can do cheaply, and the group links each to the studies that support it. Fewer dependencies reduces exposure to every risk at once; prompt updates address the lag measured in the literature; install-time restrictions and name checks address the patterns found in malicious packages. The list becomes a checklist in the starter template. Each item cites its supporting studies.
37.6 Presentation
The group presents a balanced view, separating strong findings from tentative ones.
Strong findings are those supported by several studies and designs; tentative ones rest on a single study or a narrow sample. The group gives each finding a short label and explains why. Grace asks how the conclusions would change for a project much larger than CampusEats, and the group answers that most would still hold, though the effort needed would grow. The slides mark each finding with its label, so the audience can see at a glance which conclusions are firm.
Reading critically
For each study the group asks: what question, what data, what method, what results, what limits, and what it means for a small project. Answering the same questions for every study makes comparison easier and stops the group from being swayed by impressive numbers without context. They also check whether each claim they plan to present is actually supported by the study's own results.
Selection criteria
Studies were chosen for relevance, peer review and clear methods.
Worked example
A classmate argues the studies focus on one ecosystem (npm) too much. The group agrees and notes that other ecosystems may differ, recommending readers check for studies of their own ecosystem.
Synthetic case. Seminar discussion is synthetic; the cited studies are real.
Errors and limits
Reviews can be biased by which studies are chosen; state selection criteria.
Exercises
- EX109 — Design types. What three design types were reviewed? Rubric: 3 = case collections, measurement studies, interview study; 2 = two; 1 = one; 0 = none.
- EX110 — Synthesis. Summarise the synthesis. Rubric: 3 = risk concentrated; updates lag; malicious exploit trust/install; maintainers vary; 2 = partial; 1 = one; 0 = none.
- EX111 — Ecosystem critique. What critique did the classmate raise? Rubric: 3 = too focused on npm; 2 = partial; 1 = vague; 0 = none.
Synthesis
Review evidence by design and limits, then synthesise carefully for practice.
Continue studying — chapter 37 of 40
← Chapter 36 (draft) · Course chapter index · Chapter 38 (draft) →