◇ OpenCyber Academy
Technologist · Chapter 9 · English

Chapter 9 — Evidence and measurement

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 9 — Evidence and measurement

Lesson map

Lesson 09 published. Map, example, EX025.

flowchart TD
  N0["Learning outcomes"]
  N1["9.1 Provenance"]
  N2["9.2 Observation and conclusion"]
  N3["9.3 Outcome counting"]
  N4["9.4 Negative results"]
  N5["9.5 Timeliness"]
  N6["9.6 Keeping evidence"]
  N7["9.7 Simple dashboards"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX025"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["9.1 Provenance"]
  N2["9.2 Observation and conclusion"]
  N3["9.3 Outcome counting"]
  N4["9.4 Negative results"]
  N5["9.5 Timeliness"]
  N6["9.6 Keeping evidence"]
  N7["9.7 Simple dashboards"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX025"]

[Learning outcomes]
    |
    v
[9.1 Provenance]
    |
    v
[9.2 Observation and conclusion]
    |
    v
[9.3 Outcome counting]
    |
    v
[9.4 Negative results]
    |
    v
[9.5 Timeliness]
    |
    v
[9.6 Keeping evidence]
    |
    v
[9.7 Simple dashboards]
    |
    v
[Worked example] --> [Exercise EX025]

Learning outcomes

You will (1) record the provenance of each intelligence item; (2) separate what was observed from what was concluded; (3) count outcomes rather than activity; (4) report negative results with their coverage.

9.1 Provenance

For every item Lena records who sent it, when, how, under which TLP, and what the sender based it on. Without provenance, the same indicator copied by three newsletters looks like three confirmations.

Provenance answers the question of how much weight an item deserves. If three newsletters all repeat one MCX post, the indicator has one source, not three, and counting it three times would overstate confidence. The worked example below shows the method on exactly that case. How the item arrived matters too, since an email from an unknown address claiming to be MCX deserves checking before it is trusted.

9.2 Observation and conclusion

"The proxy logged ws-114 contacting pay-portal.example at 08:03" is an observation. "A staff member probably entered credentials" is a conclusion. Reports keep them in separate sentences so readers can judge each.

The observation can be checked by anyone with access to the proxy log, while the conclusion depends on what the user did on the page, which the log does not show. Keeping them apart lets a reader accept the first and question the second. It also keeps a manager from treating a guess as a fact when deciding on a password reset.

9.3 Outcome counting

"500 indicators imported" counts activity. "Two staff were stopped from reaching the fraud page by the DNS block" counts an outcome. Lena reports both but leads with outcomes in the monthly summary.

The number of indicators imported says nothing about whether anyone was protected, and a large number may only mean a noisy feed. Two blocked visits can be checked against the DNS log and named in the summary, which a raw import count never can. Leading with outcomes keeps the monthly summary about value rather than volume.

9.4 Negative results

"No matches" is useful only with its query, time range, data source and coverage. "No DNS lookups of the three domains in the last 30 days across all staff resolvers; library resolvers not included" lets the reader see what was and was not checked.

The example sentence names its gap explicitly: library resolvers were not searched. A reader then knows that a library computer could still have looked up the domains. The worked example below shows the gap mattering, when a widened search found one lookup on a library computer that the narrower search had missed.

9.5 Timeliness

Record when intelligence arrived and when it was acted on. Long gaps point to process problems.

For an MCX item about a fraud domain, the time between its arrival and the DNS block is what decides how many residents or staff could still reach the page. Recording both times for every item lets Lena find where delays occur, such as waiting for an owner to reply or a block request sitting in a queue. The median time to action in the monthly table is calculated from these two recorded times for every item that led to action.

9.6 Keeping evidence

Exported log extracts used in a finding are saved with the ticket, hashed, and kept under the retention rule. If the finding is challenged, the evidence can be shown exactly as it was.

Hashing the extract at the time it is saved means a later copy can be compared to show it has not changed. Keeping it with the ticket keeps the evidence next to the decision it supported. The retention rule sets how long the extract is kept, so evidence is neither deleted early nor held longer than the council's rules allow.

9.7 Simple dashboards

A one-page monthly table — items received, items acted on, blocks that stopped connections, median time to action — gives the manager a view of value without overwhelming detail.

Each column answers a question the manager might ask: how much arrived, how much led to action, whether any action prevented harm, and how fast the team works. Keeping it to a single page means the manager actually reads the table. Lena keeps the detailed records behind it, so any figure can be traced to the items that produced it.

Worked example

Three newsletters list the same fraud domain. Lena traces them: all cite one MCX post. She records one independent source. Her DNS search shows one lookup by a library computer, outside the staff resolver she usually checks — so her report notes that she widened the search to library resolvers after a colleague's question.

Synthetic case. The sources and lookup are invented.

Errors and limits

Outcomes rarely prove intelligence alone made the difference; describe contribution.

Exercises

Synthesis

Record provenance, separate observation from conclusion, lead with outcomes and give coverage with every negative result.

Continue studying — chapter 09 of 30

← Chapter 08 (draft) · Course chapter index · Chapter 10 (draft) →