◇ OpenCyber Academy
Technologist · Chapter 9 · English

Chapter 9 — Evidence and measurement

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 9 — Evidence and measurement

Lesson map

Lesson 09 published. Map, example, EX025.

flowchart TD
  N0["Learning outcomes"]
  N1["9.1 What counts as evidence"]
  N2["9.2 Timestamps and sources"]
  N3["9.3 Simple measures"]
  N4["9.4 Baselines and trends"]
  N5["9.5 Misleading numbers"]
  N6["9.6 Storing evidence"]
  N7["Measurement plan"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX025"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["9.1 What counts as evidence"]
  N2["9.2 Timestamps and sources"]
  N3["9.3 Simple measures"]
  N4["9.4 Baselines and trends"]
  N5["9.5 Misleading numbers"]
  N6["9.6 Storing evidence"]
  N7["Measurement plan"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX025"]

[Learning outcomes]
    |
    v
[9.1 What counts as evidence]
    |
    v
[9.2 Timestamps and sources]
    |
    v
[9.3 Simple measures]
    |
    v
[9.4 Baselines and trends]
    |
    v
[9.5 Misleading numbers]
    |
    v
[9.6 Storing evidence]
    |
    v
[Measurement plan]
    |
    v
[Worked example] --> [Exercise EX025]

Learning outcomes

You will (1) collect evidence about system state; (2) define simple measures; (3) avoid misleading numbers.

9.1 What counts as evidence

Evidence is a record that shows the state of a system at a time: a configuration export, a log entry, an inventory file. A colleague saying "it is fine" is not evidence. Good evidence can be checked by someone else without trusting its collector. A settings export from client-b, saved with its collection time, lets Kwame confirm that audit logging was off without taking Priya's word for it. Her inventory output, her flow list and later the audit summary all become evidence in this sense, which is why each lab prints results that can be saved unchanged.

9.2 Timestamps and sources

Each piece of evidence needs when it was collected and from where, so others can check it. A settings export without a time cannot show whether a setting was wrong before or after a change, and one without a source cannot show which cluster it describes. Priya names each evidence file with the cluster, the type of export and the collection time, and notes whether it came from a real system with read-only access or from a synthetic copy. That habit later lets her compare results fairly across several weeks.

9.3 Simple measures

Useful measures for Cobalt: share of workloads using approved images, number of baseline failures per cluster, and time to answer an advisory. Each of the three measures links back to the placement trigger. The share of approved images shows how much of the inventory problem remains, baseline failures per cluster show configuration health, and time to answer an advisory measures the very question that took six hours. Priya defines each one precisely, for example counting a workload as approved only if its full image reference is on the approved list.

9.4 Baselines and trends

A single number means little; comparing it over time shows whether things improve. Seven baseline failures across three clusters means little on its own; it becomes useful when the same check runs again later and the number moves. Priya records the first values as a baseline and repeats each measurement on the same definition. If a definition must change, she notes the change beside the trend so the comparison is not quietly broken. Without that note, a falling count could simply reflect a narrower check rather than real improvement.

9.5 Misleading numbers

A percentage hides small totals. "50% of clusters failing" may mean one of two. Priya always shows counts alongside percentages. The risk is real at Cobalt's scale, where there are three clusters in the baseline lab and eight workloads in the inventory lab. One cluster changing state moves a percentage by a third. Writing "1 of 3 clusters" next to "33%" lets readers see how fragile the figure is. Priya applies the rule in her later evaluation, where counts appear before percentages in every line of output.

9.6 Storing evidence

Evidence is kept in the configuration repository with read-only history so it cannot be quietly changed. Storing evidence in the repository means each file has a commit time and author, and any later edit appears in history. That makes quiet changes visible rather than impossible, so access to rewrite history is restricted. Evidence that contains internal details, such as scanner reports, stays in the repository rather than in chat messages or personal folders, where it would be hard to find and easy to lose.

Measurement plan

Priya writes a measurement plan stating each measure, how it is collected, how often and by whom. Writing it down before collecting stops her from changing the definition later to make results look better. Kwame reviews the plan and suggests one change: measure the advisory time with a question the team has not seen before, so preparation cannot inflate the result. Priya adds that rule to the plan.

Evidence quality

Priya rates evidence as strong when it is exported directly from a system, and weaker when it is a manual note. Strong evidence is preferred for audits and comparisons.

Worked example

Priya records the six-hour advisory search as the first measurement, with date and method. She proposes measuring the same question again after the inventory work, so improvement can be shown with numbers.

Synthetic case. Measures are synthetic.

Errors and limits

Measuring what is easy rather than what matters creates false comfort.

Exercises

Synthesis

Collect timed, sourced evidence and measure trends with counts.

Continue studying — chapter 09 of 30

← Chapter 08 (draft) · Course chapter index · Chapter 10 (draft) →