Chapter 15 — Evidence and measurement

Lesson map

Lesson 15 published. Map, example, EX043.

flowchart TD
  N0["Learning outcomes"]
  N1["15.1 Defining measures"]
  N2["15.2 Collecting evidence reliably"]
  N3["15.3 Presenting honestly"]
  N4["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4
  N4 --> EX["Worked example"]
  EX --> Q["Exercise EX043"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["15.1 Defining measures"]
  N2["15.2 Collecting evidence reliably"]
  N3["15.3 Presenting honestly"]
  N4["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX043"]

[Learning outcomes]
    |
    v
[15.1 Defining measures]
    |
    v
[15.2 Collecting evidence reliably]
    |
    v
[15.3 Presenting honestly]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX043]

Learning outcomes

You will (1) define useful security measures; (2) collect evidence reliably; (3) present measurements honestly.

15.1 Defining measures

A measure is useful if it informs a decision. Sofia proposes four for the voucher system: percentage of issued vouchers matched to approved shifts, number of reconciliation mismatches per month, time to restore a validator from backup, and number of privileged actions without a second approver. Each has a target agreed with the board. She avoids vanity measures, such as counting how many tests exist without asking whether they test anything important. Dr Asante adds that a measure should be easy to explain in one sentence, or the board will not use it when it matters.

15.2 Collecting evidence reliably

Evidence must be trustworthy. Sofia collects ledger data directly from a validator rather than from screenshots, records the time and source of each extract, and stores a SHA-256 hash of each file so later changes can be detected. Database extracts are taken through the read-only reporting account. She writes the steps as a short procedure, so another volunteer could repeat them and obtain the same results. Reliable evidence matters because decisions, and sometimes disputes between members, depend on it. Evidence that cannot be repeated or verified is easily challenged and rarely persuades anyone.

15.3 Presenting honestly

Sofia's first monthly report shows that 99.6 percent of vouchers matched approved shifts, with 22 unmatched out of 5,480. She explains that 20 were late approvals later confirmed and 2 were a coordinator's data-entry error, now corrected. She shows the counts, not just the percentage, and notes that one month is too short to show a trend. Honest presentation includes uncertainty and explanations. The board appreciates that she reports the error openly. Trust in security reporting grows when reports include bad news alongside good, clearly and without blame.

Worked example

Sofia reports 22 unmatched of 5,480 vouchers (99.6 percent matched), explaining 20 late approvals and 2 entry errors, from hashed extracts.

Synthetic case. Measures and figures are synthetic.

Errors and limits

Counting tests or alerts without asking what they detect produces vanity measures.

Exercises

• EX043 — Measure example. Name one of Sofia's four measures.
  Show marking criteria

  Rubric: 3 = percentage of vouchers matched to approved shifts;; 2 = restore time;; 1 = number of tests;; 0 = none.
• EX044 — Integrity of extracts. How does she make extract changes detectable?
  Show marking criteria

  Rubric: 3 = store a SHA-256 hash of each file;; 2 = hashes;; 1 = screenshots;; 0 = none.
• EX045 — Unmatched explanation. How were the 22 unmatched vouchers explained?
  Show marking criteria

  Rubric: 3 = 20 late approvals and 2 entry errors;; 2 = late approvals;; 1 = theft;; 0 = none.

Synthesis

Decision-focused measures, reliable collection and honest reporting make evidence useful.

Continue studying — chapter 15 of 40

← Chapter 14 (draft) · Course chapter index · Chapter 16 (draft) →