Chapter 15 — Evidence and measurement
Lesson map
Lesson 15 published. Map, example, EX043.
flowchart TD
N0["Learning outcomes"]
N1["15.1 Activity versus outcome"]
N2["15.2 Measures for CampusEats"]
N3["15.3 Collecting evidence"]
N4["15.4 Baselines"]
N5["15.5 Presenting"]
N6["15.6 Honesty"]
N7["Using measures to decide"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX043"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["15.1 Activity versus outcome"]
N2["15.2 Measures for CampusEats"]
N3["15.3 Collecting evidence"]
N4["15.4 Baselines"]
N5["15.5 Presenting"]
N6["15.6 Honesty"]
N7["Using measures to decide"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX043"]
[Learning outcomes]
|
v
[15.1 Activity versus outcome]
|
v
[15.2 Measures for CampusEats]
|
v
[15.3 Collecting evidence]
|
v
[15.4 Baselines]
|
v
[15.5 Presenting]
|
v
[15.6 Honesty]
|
v
[Using measures to decide]
|
v
[Worked example] --> [Exercise EX043]
Learning outcomes
You will (1) choose measures that show whether security is improving; (2) collect evidence reliably; (3) distinguish activity from outcomes; (4) present measurements honestly.
15.1 Activity versus outcome
"We ran the scanner 400 times" is activity. "Secrets reaching the public repository fell from 2 to 0 per term" is an outcome. Outcomes show whether risk changed.
Activity measures are still useful for checking that a control is running, but they cannot show that it works. Lena keeps both kinds and labels them, so nobody mistakes a busy scanner for a safer project. Her term report puts the 400 scanner runs under control health and the fall from 2 secrets to 0 under risk outcomes.
15.2 Measures for CampusEats
Time to fix high-severity advisories, number of secrets found after push (should be zero), share of dependencies with an owner, and failed integrity checks investigated.
Each measure has a definition, a source and an owner. For example, time to fix runs from the advisory's publication to the deployed update, and the dependency owners come from the list started in the first chapter. Lena keeps the list short so the society can review it in a few minutes each month. Each has a target.
15.3 Collecting evidence
Measures come from tools automatically where possible: CI logs, scanner results, advisory tracker. Manual counts are checked by a second person.
Automatic collection avoids both effort and the temptation to round numbers in a favourable direction. Where a measure must be counted by hand, such as whether an integrity failure was investigated, the counter records how each case was judged, so the second person can check the decision rather than just the arithmetic. Lena keeps the raw logs for a year so any figure can be rechecked later.
15.4 Baselines
Before changes, Lena records a baseline: last term, 2 secrets reached the public repository and median advisory fix time was 21 days.
Without a baseline, an improvement cannot be shown and a decline cannot be noticed. Lena also records how each baseline figure was obtained, since a different method next term would make the comparison meaningless. Small numbers like two secrets per term vary by chance, so she plans to compare over several terms before drawing firm conclusions. She stores the baseline with the method used.
15.5 Presenting
A one-page summary per term with trends and short explanations.
The summary opens with the outcome measures, then the activity measures, and then any changes in how figures were collected. Each trend has a sentence explaining it: for example, a rise in advisories fixed may reflect more advisories published rather than better work. The society's committee reads it at the start of each term, and Grace receives a copy so that the security office can see progress across student projects.
15.6 Honesty
If a measure gets worse, it is reported with a reason and plan.
A summary that only ever improves is not believable. When the median fix time rose one term because a major upgrade was delayed, Lena reported it with the cause and the plan to finish the upgrade. Readers trusted the good figures more because the bad one was shown, and the committee agreed extra time for the upgrade work. Grace later used the same summary as an example for other student projects at the university.
Using measures to decide
Measures guide decisions. When the median fix time stayed high for one package family, the society looked closer and found those packages were maintained by a single volunteer who rarely released. The team decided to replace them. Without the measure, the problem would have stayed hidden in the overall average. Lena notes that good measures lead to questions, and questions lead to better decisions.
Tools for evidence
A small script gathers all measures into one table each term, so collection takes minutes rather than hours.
Audience
Results are shared with the society and Grace each term.
Worked example
After one term with the pre-commit hook and server-side scanner, no secrets reached the public repository and the median fix time fell to 9 days. Lena notes that the society also gained two experienced members that term, which may explain part of the improvement.
Synthetic case. Measures are synthetic.
Errors and limits
Zero incidents in one term may be luck; keep measuring over longer periods.
Exercises
- EX043 — Activity vs outcome. Give one activity and one outcome measure. Rubric: 3 = scanner runs; secrets reaching repo; 2 = partial; 1 = vague; 0 = none.
- EX044 — Baseline. What was the baseline? Rubric: 3 = 2 secrets; 21-day median fix; 2 = one; 1 = vague; 0 = none.
- EX045 — Other explanation. What else may explain the improvement? Rubric: 3 = two experienced members joined; 2 = partial; 1 = vague; 0 = none.
Synthesis
Measure outcomes from a baseline and report honestly, including other explanations.
Continue studying — chapter 15 of 40
← Chapter 14 (draft) · Course chapter index · Chapter 16 (draft) →