Chapter 37 — Evidence review seminar | Bachelor's Threat intelligence and OSINT

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 37 published. Map, example, EX109.

flowchart TD
  N0["Learning outcomes"]
  N1["37.1 The seminar"]
  N2["37.2 ICD 203 standards"]
  N3["37.3 Evidence versus assertion"]
  N4["37.4 Alternatives"]
  N5["37.5 Constructive critique"]
  N6["37.6 Commercial context"]
  N7["Leading the discussion"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX109"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["37.1 The seminar"]
  N2["37.2 ICD 203 standards"]
  N3["37.3 Evidence versus assertion"]
  N4["37.4 Alternatives"]
  N5["37.5 Constructive critique"]
  N6["37.6 Commercial context"]
  N7["Leading the discussion"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX109"]

[Learning outcomes]
    |
    v
[37.1 The seminar]
    |
    v
[37.2 ICD 203 standards]
    |
    v
[37.3 Evidence versus assertion]
    |
    v
[37.4 Alternatives]
    |
    v
[37.5 Constructive critique]
    |
    v
[37.6 Commercial context]
    |
    v
[Leading the discussion]
    |
    v
[Worked example] --> [Exercise EX109]

Learning outcomes

You will (1) critique a published intelligence report; (2) separate evidence from assertion; (3) assess analytic standards using ICD 203; (4) give constructive peer critique.

37.1 The seminar

Each student brings a public vendor or government CTI report and leads a 20-minute critique. The class asks: what is claimed, what evidence supports it, how confident should we be, and what is missing?

Amara chooses a government report on ransomware in health care, because its subject is close to her internship. She reads it twice, marks every claim with its evidence and drafts questions for the class. The format keeps critique focused on what readers can verify, rather than on the report's reputation or style.

37.2 ICD 203 standards

ICD 203 lists analytic tradecraft standards, including describing source quality, expressing uncertainty, distinguishing intelligence from assumptions and judgments, considering alternatives, and being relevant and logically argued. They make a useful checklist for any report.

Amara uses the standards as a checklist for her critique, marking each as met, partly met or not met with an example. The report she reviews describes sources clearly but rarely separates assumptions from judgments. She suggests how the main judgment could be rewritten to make its assumptions explicit. The class agrees that the checklist makes critique more consistent between students.

37.3 Evidence versus assertion

A report saying "the actor is state-sponsored" without citing evidence is asserting. One that lists infrastructure overlap, tooling and timing — and states confidence — is arguing.

Amara marks each attribution claim in her report as asserted or argued. Most claims about the actor's motives are asserted, while claims about techniques are well argued with indicators and timing. She explains that asserted claims are not necessarily wrong, but a reader cannot check them and should treat them as the author's opinion until evidence appears.

37.4 Alternatives

Did the report consider other explanations? A false-flag possibility, a shared tool sold to many criminals, or coincidence?

Amara's chosen report considers no alternatives, so she proposes two: a criminal affiliate using leaked tooling, and coincidence in timing between two unrelated intrusions. For each, she lists what evidence would support or rule it out. The class discusses whether the report's conclusion would survive these tests. Considering alternatives strengthens a conclusion that survives them and exposes one that does not. The class adopts this habit.

37.5 Constructive critique

Start with what the report does well; then name specific gaps; propose improvements. Critique the work, not the authors.

Amara opens her critique by noting the report's clear timeline and good use of public advisories. She then names two specific gaps, with page references, and proposes improvements for each. Classmates follow the same structure in their feedback to her, which keeps the discussion respectful and useful. Her tutor notes that specific critique is more helpful than general praise or general criticism.

37.6 Commercial context

Vendor reports also market products. That does not make them wrong, but it shapes what is emphasised — threats their product detects, dramatic names, large numbers. Note the publisher's interest alongside the evidence quality, and look for independent corroboration of headline figures.

Amara notes the publisher's product focus in her critique. Her report's headline figure for affected organisations comes from the vendor's own customer telemetry, which may not represent all victims. She looks for the same figure in a government source and finds a smaller estimate based on reported incidents. Presenting both with their sources lets readers see why they differ.

Leading the discussion

The presenter prepares three questions for the class and a one-slide summary of the report's key claims. Timekeeping matters: leave half the slot for discussion, and end by agreeing which parts of the report the class would trust for decisions.

Worked example

Amara reviews a public report attributing hospital attacks to a named group. It shows overlapping infrastructure but does not discuss shared hosting, and it uses "almost certainly" without explaining the evidence weight. She notes the report is useful for its technique list (T1566, T1486) but rates its attribution confidence as low pending more evidence.

Synthetic case. The report is invented.

Errors and limits

Public reports omit restricted evidence; absence of evidence in the text does not prove absence of evidence altogether.

Exercises

EX109 — Standards list. Name three ICD 203 analytic standards.
Show marking criteria
Rubric: 3 = three valid (source quality, uncertainty, distinguish assumptions/judgments, alternatives, relevance, logic); 2 = two; 1 = one; 0 = none.
EX110 — Alternative explanation. Suggest an alternative to "shared infrastructure means the same group".
Show marking criteria
Rubric: 3 = shared hosting/tools sold widely/coincidence; 2 = vague; 1 = minimal attempt; 0 = none.
EX111 — Missing evidence caution. Why should a reviewer be careful about concluding a public report "has no evidence"?
Show marking criteria
Rubric: 3 = restricted evidence may be withheld; absence in text ≠ absence; 2 = partial; 1 = not careful; 0 = none.

Synthesis

Read reports against standards, demand evidence, seek alternatives, note interests and critique constructively.

Continue studying — chapter 37 of 40

← Chapter 36 (draft) · Course chapter index · Chapter 38 (draft) →