OpenCyber Academy
Bachelor's · Chapter 15 · English
Course code: OCA-110Chapter 15: OCA-110-C015

Chapter 15 — Evidence and measurement

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 15 published. Map, example, EX043.

flowchart TD
  N0["Learning outcomes"]
  N1["15.1 Three layers"]
  N2["15.2 Provenance"]
  N3["15.3 Measuring intelligence"]
  N4["15.4 Feedback"]
  N5["15.5 Timeliness"]
  N6["15.6 Precision and recall for blocks"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX043"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["15.1 Three layers"]
  N2["15.2 Provenance"]
  N3["15.3 Measuring intelligence"]
  N4["15.4 Feedback"]
  N5["15.5 Timeliness"]
  N6["15.6 Precision and recall for blocks"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX043"]

[Learning outcomes]
    |
    v
[15.1 Three layers]
    |
    v
[15.2 Provenance]
    |
    v
[15.3 Measuring intelligence]
    |
    v
[15.4 Feedback]
    |
    v
[15.5 Timeliness]
    |
    v
[15.6 Precision and recall for blocks]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX043]

Learning outcomes

You will (1) distinguish observation, inference and assessment; (2) record provenance for each piece of evidence; (3) measure the usefulness of intelligence; (4) separate activity metrics from outcome metrics.

15.1 Three layers

An observation: "the proxy logged a connection to portal-login.example at 09:41". An inference: "the user probably clicked the phishing link". An assessment: "HCHT was likely targeted by the sector campaign". Each layer needs its own support, and reports should label which is which.

Amara marks each sentence in her drafts as an observation, inference or assessment. Readers can then see which statements are facts from logs and which are judgments, and they can challenge the judgments without doubting the facts. When new evidence arrives, an inference may change while the observation stays the same.

15.2 Provenance

For every item record who reported it, when, through which channel, under which TLP, and what it is based on. Without provenance, an indicator copied between feeds appears to have many sources when it has one — circular reporting.

Amara's database stores provenance for every indicator, and her reports cite the original source rather than the feed that repeated it. When two HSSC members report the same domain, she checks whether both saw it independently or one copied the other's report. Only independent sightings raise confidence. Circular reporting has twice inflated confidence in HSSC summaries, which is why the community now asks members to state their original source.

15.3 Measuring intelligence

Counting indicators ingested measures activity. Outcome measures ask: did intelligence lead to a block that stopped something, a hunt that found something, or a decision that changed? Fewer, better indicators can produce more value.

Amara adds an outcome column to the indicator database: blocked, hunted with a result, hunted without a result, or no action. After a quarter, she can show which sources led to action and which only added volume. The figures help the team decide which feeds are worth the processing time and which could be dropped.

15.4 Feedback

Consumers should tell producers what helped. HSSC's monthly feedback form asks members which items led to action. Amara fills it in honestly, including "no action needed".

"No action needed" is useful feedback, because it tells HSSC that an item did not apply to members like HCHT. Amara adds a sentence explaining why, such as "we do not run the affected product". HSSC shares a summary of all members' feedback each quarter, which shows which report types members find most useful.

15.5 Timeliness

An indicator delivered after the campaign ended has little value. Record the time between first sighting and HCHT action.

Amara records the first-sighting time from the source report and the time HCHT applied a block or completed a hunt. The median gap is reported each quarter. A long gap may come from a slow feed, a slow review or a slow change process, and the record shows which step took longest. Shortening the slowest step is the most effective improvement.

15.6 Precision and recall for blocks

Precision asks what share of blocked items were truly malicious; recall asks what share of malicious items were blocked. A feed with high recall but poor precision floods the service desk with complaints; one with high precision but low recall misses campaigns. Track both rather than a single "hits" count.

Amara measures both for the main feeds each quarter. Precision is checked by sampling blocked items and confirming each one; recall is harder, so she estimates it from incidents and HSSC reports that later proved malicious. Both figures are reported with their counts and their limits.

Worked example

Three feeds report the same domain. Amara checks provenance: two cite the third. She records one independent source, not three, and her confidence stays moderate. Later she logs that blocking the domain stopped two clicks — an outcome metric for the quarterly review.

Synthetic case. The feeds and clicks are invented.

Errors and limits

Outcomes are hard to attribute to intelligence alone. Report contribution, not proof.

Exercises

Synthesis

Separate evidence layers, keep provenance, measure outcomes and feed back to producers.

Continue studying — chapter 15 of 40

← Chapter 14 (draft) · Course chapter index · Chapter 16 (draft) →

Worked answers

EX043. Inference — it goes beyond the logged connection to infer a user action.

EX044. Two of the three feeds cited the third, so there was a single independent source and confidence did not rise.

EX045. Activity. An outcome alternative is the number of blocked connections that prevented credential entry or the number of hunts that found real activity.