OpenCyber Academy
Bachelor's · Chapter 37 · English
Course code: OCA-114Chapter 37: OCA-114-C037

Chapter 37 — Evidence review seminar

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 37 published. Map, example, EX109.

flowchart TD
  N0["Learning outcomes"]
  N1["37.1 A ladder of evidence"]
  N2["37.2 Reading a measurement paper"]
  N3["37.3 Reading a survey"]
  N4["37.4 Structured peer review"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX109"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["37.1 A ladder of evidence"]
  N2["37.2 Reading a measurement paper"]
  N3["37.3 Reading a survey"]
  N4["37.4 Structured peer review"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX109"]

[Learning outcomes]
    |
    v
[37.1 A ladder of evidence]
    |
    v
[37.2 Reading a measurement paper]
    |
    v
[37.3 Reading a survey]
    |
    v
[37.4 Structured peer review]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX109]

Learning outcomes

You will (1) critically review a published OT security claim; (2) grade evidence quality from anecdote to replicated measurement; (3) check whether conclusions follow from the data presented; and (4) give and receive structured peer review.

37.1 A ladder of evidence

From weakest to strongest: vendor marketing claim; single anecdote; case report from a primary source; systematic measurement on one system; measurement replicated across systems or by independent teams. A seminar asks of every claim: which rung is this, and does the conclusion overreach it?

Maya's review of a classmate's report applies the ladder: "CUSUM eliminates false alarms in district heating" claims a replicated result, but the evidence is one systematic measurement on synthetic data, 0 of 40 runs. That rung supports "no false alarms observed, upper bound about 8.8 %", not elimination, and says nothing about real plants.

37.2 Reading a measurement paper

The Mirai study by Antonakakis et al. (USENIX Security 2017) is a good seminar text: it combines several data sources — network telescope scans, honeypots, DNS data and attack victim data — to estimate infection size and behaviour. Ask: what does each source see and miss? How were infected devices counted when IP addresses change? Which conclusions depend on one source only?

For LDHP's seminar, the useful exercise is to take one conclusion, such as the share of infections through default Telnet credentials, and name the source it rests on.

37.3 Reading a survey

Giraldo et al. (ACM Computing Surveys, 2018) classify physics-based detection work. A survey's strength is breadth; its risk is that it reports what papers claim rather than re-testing them. Use it to find primary papers and recurring weaknesses, such as inconsistent attacker models.

For Maya's detector study, Giraldo et al. pointed to Urbina et al.'s impact-based evaluation, which shaped her decision to vary the attack slope instead of reporting one detection rate. The survey also warned her that many detector papers use no consistent attacker model, which is why her pre-registration names the 0.08 °C-per-sample ramp as her attack class before any run.

37.4 Structured peer review

Review format: summary in your own words; main strengths; major concerns (validity, missing comparisons, overreach); minor issues; a recommendation. Comment on the work, not the person.

Maya's review of her classmate's CUSUM report follows this order exactly. Her summary restates the claim, her strength names the reproducible script with seeds, her major concern is that 0 of 40 synthetic runs cannot support "eliminates", and her minor issue is the delay given in samples rather than minutes. The recommendation, to revise the claim and add the bound of about 8.8 %, tells the author what to change. Separating major from minor concerns shows which change is essential before the report can stand.

Worked example (synthetic LDHP)

A classmate's report claims "CUSUM eliminates false alarms in district heating". Maya's review: summary; strength — reproducible script with seeds; major concern — 0/40 synthetic runs support "no false alarms observed, upper bound ≈ 8.8 %", not elimination, and nothing was tested on real plant data; minor — the delay unit (samples) is not converted to minutes; recommendation — revise the claim and add the bound.

Synthetic case — Lakeside District Heating Plant. The classmate's report is invented.

Errors and limits

Peer review checks reasoning, not truth; reviewers miss errors too. Prestige of a venue is not evidence of correctness.

Exercises

Synthesis

Evidence review asks how strong the evidence is, whether the conclusion fits it, and how to say so constructively.

Continue studying — chapter 37 of 40

← Chapter 36 (draft) · Course chapter index · Chapter 38 (draft) →

Worked answers

EX109. (a) Bottom rung — an unverified marketing claim with no method or denominator. (b) A primary case report from a government source: stronger, but a single incident.

EX110. The estimate of how many devices were infected: honeypots only observe bots that happen to scan or attack them, so they undercount and are biased by where the honeypots sit; the telescope data provide the broader population view.

EX111. "The detector is tuned and evaluated on the same simulated plant, so the reported false-alarm and detection rates are likely optimistic and may not generalise. Please tune thresholds on separate seeds, freeze them, and evaluate on at least one additional process model or published dataset."