Chapter 15 — Evidence and measurement
Lesson map
Lesson 15 published. Map, example, EX043.
flowchart TD
N0["Learning outcomes"]
N1["15.1 Evidence"]
N2["15.2 Measures"]
N3["15.3 Baselines"]
N4["15.4 Honest presentation"]
N5["15.5 Storage"]
N6["Collecting the baseline"]
N7["Measures and decisions"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX043"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["15.1 Evidence"]
N2["15.2 Measures"]
N3["15.3 Baselines"]
N4["15.4 Honest presentation"]
N5["15.5 Storage"]
N6["Collecting the baseline"]
N7["Measures and decisions"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX043"]
[Learning outcomes]
|
v
[15.1 Evidence]
|
v
[15.2 Measures]
|
v
[15.3 Baselines]
|
v
[15.4 Honest presentation]
|
v
[15.5 Storage]
|
v
[Collecting the baseline]
|
v
[Measures and decisions]
|
v
[Worked example] --> [Exercise EX043]
Learning outcomes
You will (1) collect evidence with time and source; (2) define measures; (3) avoid misleading numbers.
15.1 Evidence
Agent reports, exports and logs, each saved with collection date and source.
Aisha's baseline folder holds five files collected in one week: the device export, the encryption report, the update status for the last monthly release, the asset register and the server service lists. Each file name starts with the collection date and the system it came from. A reader of the dissertation can therefore trace every baseline number back to one dated file. A copy of each export is never edited; any cleaning Aisha does happens in a separate file with its own date and a note of what changed.
15.2 Measures
Encryption share, patch latency, unmanaged devices and unneeded server services.
Each measure has a fixed definition written before the first count. Encryption share is laptops reported encrypted out of 45; patch latency is days from release to install after restart; unmanaged devices come from the register comparison; unneeded services come from the role documents. At the baseline the four values are 32 of 45, 26 of 45 within 14 days, six and four. Each measure is calculated by a lab script, so Gareth can recalculate it later without relying on Aisha's spreadsheet or memory.
15.3 Baselines
The first measurements become the baseline for later comparison.
Bramley's baseline is the week Aisha collects the five evidence files, and its figures are not changed afterwards, even when errors are found later. If a mistake is discovered, she records a correction note beside the original figure instead of overwriting it. The evaluation in chapter 39 compares the final measures with these baseline figures, so the starting point must stay fixed. Dr Grant reviews the baseline method before any change is made, which stops early fixes from leaking into the starting figures.
15.4 Honest presentation
Counts with percentages; tails as well as averages.
Aisha's baseline table shows "32 of 45 (71%)" rather than "71%" alone, and it lists the ten laptops over 30 days alongside the median of 12. She also shows the two laptops that did not report as a separate row, not hidden inside either total. Gareth can see at once how many devices each percentage describes. In the board summary the same rule applies: four before-and-after measures, each shown as a count with its percentage beside it. Where a percentage rests on fewer than ten devices, Aisha gives the count alone.
15.5 Storage
Evidence kept read-only.
Aisha stores the five baseline files in a read-only folder on the university's encrypted storage, and works only on copies. She records a hash of each file on the day it is collected, using the same idea as checking an installer download. If a later figure disagrees with the baseline, the hashes show whether the evidence has changed since collection. Read-only storage also protects Aisha: if her conclusions are questioned, she can show that the evidence behind them has not been edited since collection.
Collecting the baseline
Aisha collects baseline evidence in one week: a device export from the management server, an encryption report, the update status for the last monthly release, the asset register and a list of running services on each server. Each file is named with the date and source and stored read-only. She writes a short note describing how each export was produced, so the same steps can be repeated for the evaluation in chapter 39.
She also records what is missing: two laptops did not report at all, so their status is unknown. Rather than guessing, she counts them separately. This honesty matters later, because the evaluation compares like with like.
When she presents the baseline to Gareth, she shows counts first, then percentages, then a short list of the worst cases. Gareth says it is the first time he has seen the whole fleet's state on one page.
Measures and decisions
Each measure is linked to a decision it supports: encryption share decides rollout priority; patch tails decide where to investigate; unmanaged devices decide inventory work; unneeded services decide server hardening. Measures without a decision are dropped to keep the work focused.
Worked example
Aisha's baseline table shows 32 of 45 encrypted, 26 of 45 patched within 14 days, 6 unmanaged devices and 4 unneeded server services.
Synthetic case. Synthetic.
Errors and limits
Measures changed mid-project break comparisons.
Exercises
- EX043 — Measures. List the four measures. Rubric: 3 = encryption, latency, unmanaged, services; 2 = three; 1 = two; 0 = none.
- EX044 — Baseline. Give the baseline encryption and patch figures. Rubric: 3 = 32/45, 26/45; 2 = partial; 1 = vague; 0 = none.
- EX045 — Consistency. Why keep measures fixed? Rubric: 3 = comparisons; 2 = partial; 1 = vague; 0 = none.
Synthesis
A fixed baseline makes improvement measurable.
Continue studying — chapter 15 of 40
← Chapter 14 (draft) · Course chapter index · Chapter 16 (draft) →