Chapter 37 — Evidence review seminar
Lesson map
Lesson 37 published. Map, example, EX109.
flowchart TD
N0["Learning outcomes"]
N1["37.1 Purpose"]
N2["37.2 Example claim"]
N3["37.3 Research example"]
N4["37.4 Internal data"]
N5["37.5 Presenting"]
N6["37.6 Using evidence"]
N7["Grading evidence"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX109"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["37.1 Purpose"]
N2["37.2 Example claim"]
N3["37.3 Research example"]
N4["37.4 Internal data"]
N5["37.5 Presenting"]
N6["37.6 Using evidence"]
N7["Grading evidence"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX109"]
[Learning outcomes]
|
v
[37.1 Purpose]
|
v
[37.2 Example claim]
|
v
[37.3 Research example]
|
v
[37.4 Internal data]
|
v
[37.5 Presenting]
|
v
[37.6 Using evidence]
|
v
[Grading evidence]
|
v
[Worked example] --> [Exercise EX109]
Learning outcomes
You will (1) critically review evidence used in cloud security claims; (2) assess vendor reports, studies and internal data; (3) present a critique.
37.1 Purpose
Students bring a claim from a source and assess its evidence: who made it, what data, what method, what limits.
The same questions apply to any claim Samuel might use in his thesis. Asking who made a claim and from what data separates research with a described method from marketing, and helps him decide how strongly he can rely on each source when making his own argument. Each student also states how they would use the claim if it held, so the seminar connects evidence to decisions rather than treating criticism as an end in itself.
37.2 Example claim
A vendor report states that "most cloud breaches are caused by misconfiguration". Samuel checks: the report's data come from the vendor's customers, the definition of breach is broad and the method is not described. The claim is plausible and consistent with research on configuration errors (Rahman et al., 2019), but the specific percentage is weakly supported.
Samuel's verdict is moderate support for the general idea and weak support for any particular figure. He notes that his own opening case fits the pattern, but one case cannot confirm a broad claim. In his thesis, he uses the idea with a hedge and does not quote the vendor's percentage.
37.3 Research example
Shu, Gu and Enck (2017) analysed real Docker Hub images with a described method; their findings are stronger evidence for image vulnerability prevalence at that time.
The paper's method is described, its data can be checked and its limits are stated, which is why it scores higher than the vendor report. Its age is the main weakness, since public images and scanning practices have changed since. Samuel cites it as evidence of the problem at that time, consistent with his own base image study.
37.4 Internal data
Samuel's own studies are reviewed by classmates, who point out the short time frames.
Classmates noted that his base image study covered one month and his namespace template study one term. Samuel accepts the criticism and states both time frames prominently in his thesis. He also notes that repeating both studies next year, using the scripts he has kept, would test whether the findings hold. He also accepts a point about reviewer knowledge in the template study, which his blinded relabelling partly addresses. Reviewing his own work with the same grid he applies to others keeps his verdicts consistent, and the scores show where his studies are strongest and weakest.
37.5 Presenting
Each student gives a ten-minute critique with a verdict: strong, moderate or weak support.
Ten minutes is enough to state the claim, the evidence, the verdict and the reasons. A clear verdict helps the audience compare critiques, and the reasons make the verdict open to challenge. Samuel's critique of the vendor report followed exactly that structure and drew questions only about the grid scores. Each critique is followed by five minutes of discussion, and the presenter records any change to their verdict. Classmates asked whether moderate support for the general idea was generous; after he showed the configuration-error research beside it, the verdict stayed as it was, with weak support for the figure.
37.6 Using evidence
Weak evidence can still guide action if the action is cheap and sensible; it should not be quoted as fact.
For example, the vendor claim, even weakly supported, points in the same direction as the opening case and the research. Acting on it by checking configuration regularly is cheap and sensible. Quoting its percentage in a report to leadership would overstate what is known. Samuel's thesis follows this rule by separating claims he relies on, which have strong evidence, from claims that motivated cheap actions, which are clearly labelled as weaker. Readers can see which conclusions would change if a weak source turned out to be wrong.
Grading evidence
The seminar uses a simple grid: source type, data described, method described, sample size, conflicts of interest and age. Each criterion scores 0 to 2. The grid does not replace judgement but makes it explicit and comparable. Samuel's grid for the vendor report scores 4 of 12, for Shu et al. 10 of 12 (losing points mainly for age), which matches the seminar's verdicts.
Reflection
Samuel notes his own studies would score about 7 of 12, mainly for short time frames.
Worked example
The seminar agrees that the vendor claim should be cited as "a vendor survey reported" rather than as fact. Samuel updates his thesis wording accordingly.
Synthetic case. Seminar content is synthetic; cited papers are real.
Errors and limits
Even peer-reviewed studies age quickly in fast-changing cloud environments.
Exercises
- EX109 — Evidence questions. What four questions assess evidence? Rubric: 3 = who, what data, what method, what limits; 2 = three; 1 = one; 0 = none.
- EX110 — Vendor claim. How was the vendor claim judged? Rubric: 3 = plausible but weakly supported percentage; 2 = partial; 1 = vague; 0 = none.
- EX111 — Wording. How was the thesis wording changed? Rubric: 3 = "a vendor survey reported"; 2 = partial; 1 = vague; 0 = none.
Synthesis
Judge claims by their evidence, not their confidence.
Continue studying — chapter 37 of 40
← Chapter 36 (draft) · Course chapter index · Chapter 38 (draft) →