◇ OpenCyber Academy
Technologist · Chapter 9 · English

Chapter 9 — Evidence and measurement

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 9 — Evidence and measurement

Lesson map

Lesson 09 published. Map, example, EX025.

flowchart TD
  N0["Learning outcomes"]
  N1["9.1 Choosing indicators"]
  N2["9.2 Collecting evidence"]
  N3["9.3 Before and after"]
  N4["9.4 Reporting simply"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX025"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["9.1 Choosing indicators"]
  N2["9.2 Collecting evidence"]
  N3["9.3 Before and after"]
  N4["9.4 Reporting simply"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX025"]

[Learning outcomes]
    |
    v
[9.1 Choosing indicators]
    |
    v
[9.2 Collecting evidence]
    |
    v
[9.3 Before and after]
    |
    v
[9.4 Reporting simply]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX025]

Learning outcomes

You will (1) choose simple indicators; (2) collect evidence reliably; (3) compare before and after.

9.1 Choosing indicators

Tomás needs to show whether changes improve security. He chooses indicators linked to the risks: the number of zone pairs that can reach each other, the number of unapproved services on servers, the number of accounts not matching current lists, and the number of switches with insecure management protocols. Each is easy to count with a script and easy for the head teacher to understand. Good indicators move only when security actually changes, rather than when someone simply changes how they are counted.

9.2 Collecting evidence

Each indicator has a defined method: which data source, which script, which date. Tomás saves script outputs with dates in a project folder, so figures can be rechecked later. He avoids relying on memory or verbal reports, which cannot be verified. Evidence collected the same way each time can be compared reliably. When a method changes, such as using a new inventory source, he records the change and, where possible, measures once with both methods so the effect of the change itself is known.

9.3 Before and after

To show improvement, Tomás measures before changes and again afterwards. Before, all six proposed zones are effectively one, so every zone pair can reach every other: thirty ordered pairs. After implementation, he expects only the fourteen needed flows to cross between zones. He is careful to measure in similar conditions, such as during term time, because holiday periods have very different traffic. Before and after comparisons are the simplest convincing evidence that a change achieved its goal. Both measurements use the same scripts.

9.4 Reporting simply

Tomás reports indicators in a short table with the before value, the after value, the target and a one-line comment. He adds sample sizes and dates. He avoids percentages for very small counts, writing "2 of 2 switches" rather than "100 percent". The report is shared with Gareth and the head teacher monthly during the placement. Simple, honest reporting builds trust and lets the school see progress without needing technical expertise, while still giving technical readers the detail they need.

Writing 2 of 2 switches also shows the reader how little evidence lies behind the figure.

Worked example

Before values: thirty reachable zone pairs, four unapproved services, thirty-nine leftover accounts and two Telnet switches.

Synthetic case. Indicators are synthetic.

Errors and limits

Comparing a holiday week with a term week can create false improvements.

Exercises

Synthesis

Simple, reproducible indicators measured before and after show whether changes work.

Continue studying — chapter 09 of 30

← Chapter 08 (draft) · Course chapter index · Chapter 10 (draft) →