Chapter 37 — Evidence review seminar
Lesson map
Lesson 37 published. Map, example, EX074.
flowchart TD
N0["Learning outcomes and connection"]
N1["37.1 A review checklist"]
N2["37.2 Small samples and intervals"]
N3["37.4 Peer review"]
N4["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4
N4 --> EX["Worked example"]
EX --> Q["Exercise EX074"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["37.1 A review checklist"]
N2["37.2 Small samples and intervals"]
N3["37.4 Peer review"]
N4["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX074"]
[Learning outcomes and connection]
|
v
[37.1 A review checklist]
|
v
[37.2 Small samples and intervals]
|
v
[37.4 Peer review]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX074]
Learning outcomes and connection
After this seminar you can (1) critically review a security study, report or vendor claim using a structured checklist; (2) quantify the uncertainty of small-sample results with confidence intervals; (3) give and receive peer review that improves work; and (4) decide how much weight a piece of evidence deserves in your capstone. It builds on Chapters 15, 18, 27 and 33 and prepares capstone implementation and evaluation (Chapters 38–39).
37.1 A review checklist
For any study, report or claim: Question — what exactly is claimed? Data — where did it come from, how much, how representative? Method — could it be repeated; was there a comparison or control? Results — are uncertainties reported; do numbers add up? Validity — construct, internal, external, reliability (Chapter 18)? Interests — who funded or produced it, and what would they prefer to find? Limits — does the work state them, and are its conclusions within them? In security, common weaknesses include evaluation on unrealistic data, base-rate neglect, and results that do not transfer to live environments (Arp et al., 2022; Sommer and Paxson, 2010). In time-series anomaly detection, Kim and colleagues showed that a popular evaluation protocol can make even random scores look strong (Kim et al., 2022).
37.2 Small samples and intervals
Critical-service evidence often comes from few observations: five restore drills, twenty exercises, a dozen incidents. A point estimate ("85% of drills met the objective") hides how uncertain it is. A confidence interval gives a range of plausible values. Two common approaches for a proportion: the bootstrap, which resamples the observed data many times (Efron, 1979), and the Wilson score interval, a formula that behaves well for small samples and proportions near 0 or 1 (Wilson, 1927). The bootstrap is general but behaves poorly at small sizes and near boundaries — it can never produce a value above the maximum observed, so with 4 of 5 successes it may suggest an upper bound of 100%.
37.3 Worked example and Lab 14 (synthetic)
This case is synthetic. CEDC ran 20 restore drills; 17 met the 4-hour objective. A smaller operator ran 5; 4 met it.
# Lab 14 - bootstrap confidence interval for a restore-drill success rate (synthetic, seed 14, offline)
import random
random.seed(14)
drills = [1] * 17 + [0] * 3 # 20 restore drills, 17 met the 4-hour objective
obs = sum(drills) / len(drills)
boots = sorted(sum(random.choice(drills) for _ in drills) / len(drills) for _ in range(10_000))
lo, hi = boots[249], boots[9749]
print(f"observed success = {obs:.2f}; 95% bootstrap interval = [{lo:.2f}, {hi:.2f}]")
small = [1, 1, 1, 0, 1]
boots = sorted(sum(random.choice(small) for _ in small) / len(small) for _ in range(10_000))
print(f"with only 5 drills (4 met): [{boots[249]:.2f}, {boots[9749]:.2f}]")
from math import sqrt
def wilson(k, n, z=1.96):
p = k / n; d = 1 + z*z/n
c = (p + z*z/(2*n)) / d; h = z * sqrt(p*(1-p)/n + z*z/(4*n*n)) / d
return c - h, c + h
print("Wilson 95%% for 17/20: [%.2f, %.2f]; for 4/5: [%.2f, %.2f]" % (*wilson(17, 20), *wilson(4, 5)))
Expected output:
observed success = 0.85; 95% bootstrap interval = [0.70, 1.00]
with only 5 drills (4 met): [0.40, 1.00]
Wilson 95% for 17/20: [0.64, 0.95]; for 4/5: [0.38, 0.96]
Both methods say the same thing in substance: 17 of 20 is consistent with a true success rate anywhere from about two-thirds to about 95%; 4 of 5 is consistent with anything from under 40% to nearly always. The bootstrap's upper bound of 1.00 is an artefact of resampling data that contains no value above 1; the Wilson interval is the better choice here. For the board, the message is: "we have good but not conclusive evidence that restores usually meet four hours; one in four failing would not surprise us". Five drills cannot support a claim of reliability.
37.4 Peer review
Seminar groups exchange capstone evidence chapters. A useful review states what the work claims, what is convincing, the two or three most important problems (each with a suggested fix), and minor points. Review the work, not the person; be specific ("the restore-time claim rests on one drill in a test environment; can you add the drill's conditions and an interval?"). Authors respond point by point: accepted and changed, or declined with reasons. To the restore-time comment, for example: "Added the drill's conditions; an interval is not possible from one drill, so the claim is now stated as a single observation."
37.5 Common errors and limits
- Reporting a proportion without its sample size or interval.
- Treating vendor benchmarks as independent evidence.
- Reviews that only praise or only attack.
- Ignoring how evaluation protocols can inflate results.
Lab 14 shows the first: "85 %" without "of 20" and an interval hides that the true rate may be 64 %. A detector vendor's own benchmark (EX074) is the second. A review that says only "great work" helps nobody. The fourth is Kim and colleagues' point about evaluation protocols. "The restore success rate rests on Lab 14's 17 of 20" names the evidence; "the results seem optimistic" does not.
Exercises — chapter 37
EX073 — How many drills? Objective: relate sample size to precision. Inputs: the Wilson function from Lab 14. Steps: compute the Wilson 95% interval for 34/40 and 85/100 and comment on how evidence accumulates. Expected evidence: two intervals and a comment. Rubric: 3 = approximately [0.71, 0.93] and [0.77, 0.91], width shrinking roughly with the square root of n; recommends a drill programme; 2 = intervals only; 1 = one; 0 = none.
EX074 — Review a claim. Objective: apply the checklist. Inputs: a vendor brochure (synthetic): "Our OT anomaly detector achieved 99.7% accuracy on a public water-testbed dataset." Steps: apply the checklist and list four questions to ask the vendor. Expected evidence: review and questions. Rubric: 3 = accuracy misleading under imbalance (Chapter 27), testbed realism and transfer, evaluation protocol, interests; questions on precision/recall at operating threshold, live deployment results, false alarms per day, independent evaluation; 2 = partial; 1 = generic; 0 = none.
Chapter synthesis
Evidence review is structured scepticism: question, data, method, results, validity, interests and limits. Report small-sample results with intervals — Wilson for proportions — and let peer review improve work by being specific, prioritised and respectful.
Continue studying — chapter 37 of 40
← Chapter 36 · Course chapter index · Chapter 38 →
Continue to the next authored lesson.