Chapter 39 — Capstone evaluation
Lesson map
Lesson 39 published. Map, example, EX115.
flowchart TD
N0["Learning outcomes"]
N1["39.1 Measures"]
N2["Lab — before and after"]
N3["39.2 Reading the result"]
N4["39.3 Against criteria"]
N5["Before and after"]
N6["Cost of the work"]
N7["Answering the research question"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX115"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["39.1 Measures"]
N2["Lab — before and after"]
N3["39.2 Reading the result"]
N4["39.3 Against criteria"]
N5["Before and after"]
N6["Cost of the work"]
N7["Answering the research question"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX115"]
[Learning outcomes]
|
v
[39.1 Measures]
|
v
[Lab — before and after]
|
v
[39.2 Reading the result]
|
v
[39.3 Against criteria]
|
v
[Before and after]
|
v
[Cost of the work]
|
v
[Answering the research question]
|
v
[Worked example] --> [Exercise EX115]
Learning outcomes
You will (1) compare before and after measures; (2) judge success; (3) state limits.
39.1 Measures
Same as the baseline from chapter 15.
Aisha repeats exactly the measurement steps written down in chapter 15, using the same exports, definitions and counting rules. Encryption share is still counted out of 45 laptops, and unknown devices are still shown separately. Using the same method means the change between baseline and final figures reflects Bramley's fleet, not a change in how it was measured. The final exports are collected in one week, as the baseline was, and saved read-only with their dates.
Lab — before and after
"""Before and after metrics for Bramley's endpoint programme (synthetic, stdlib only)."""
m = [("laptops encrypted", 32, 45, 45, 45), ("patched within 14 days", 26, 45, 41, 45),
("unneeded server services", 4, None, 0, None), ("unmanaged devices", 6, None, 1, None)]
for name, b, bt, a, at in m:
if bt: print(f"{name}: {b}/{bt} -> {a}/{at}")
else: print(f"{name}: {b} -> {a}")
Run with python3 lab_compare.py. Verified output:
laptops encrypted: 32/45 -> 45/45
patched within 14 days: 26/45 -> 41/45
unneeded server services: 4 -> 0
unmanaged devices: 6 -> 1
39.2 Reading the result
Encrypted laptops rose from 32 to 45 of 45; patched within 14 days from 26 to 41 of 45; unneeded server services fell from 4 to 0; unmanaged devices from 6 to 1.
As shares, encryption rose from about 71% to 100% and patching within 14 days from about 58% to about 91%. The services and unmanaged figures are counts, so Aisha reports them without percentages. Each final figure comes from the same kind of export as its baseline figure, dated and saved in the evidence folder. Screen lock compliance barely moved because it was already high, and Aisha reports that too.
39.3 Against criteria
Encryption, services and unmanaged criteria met; patching at 91% meets the 90% target.
The proposal set four criteria: 100% encrypted, 90% patched within 14 days, zero unneeded services and at most one unmanaged device. Forty-one of 45 is 91.1%, just above the 90% target, which needed at least 41 laptops. The unmanaged figure of one meets its criterion only at the limit, and Aisha says so rather than presenting it as comfortably met. Aisha does not add new criteria at this stage, because criteria chosen after seeing results would not test anything.
39.4 Limits
One organisation, a few months, other changes coincided.
The evaluation covers one charity of 45 laptops over a few months, so seasonal effects and staff changes are not captured. Several changes overlapped, such as reminders and forced restarts, so individual effects cannot be separated. Some figures depend on devices reporting to the management server, which is why non-reporting devices are counted separately in every table. She proposes a full year of measurement as the most useful follow-up, to show whether the gains last.
Before and after
The evaluation lab compares the baseline with the final measures. Encrypted laptops rose from 32 to 45; laptops patched within 14 days rose from 26 to 41; unneeded server services fell from 4 to 0; unmanaged devices fell from 6 to 1. The remaining unmanaged device is the one left in the final weekly comparison, recorded with its follow-up action.
Aisha presents the numbers with their sources and dates, and repeats the measurement method from chapter 15 so the comparison is fair. Where a number depends on reporting devices, she notes how many reported.
She links each change to the action that caused it, while acknowledging that several actions overlapped, as discussed in chapter 33.
Cost of the work
Total staff time over the project was roughly 60 hours for Gareth, which the manager judges good value.
Answering the research question
Aisha's question was whether a small organisation could reach and keep a basic secure endpoint baseline with one officer. The evidence says yes, with clear routines and modest tools.
Project log entry
Aisha checks the evaluation against her original research question and success criteria from the proposal. All four criteria are met, but the unmanaged-device target only at its limit of one, which she reports plainly rather than rounding it into a clean result.
Worked example
Bramley's board asks what the stolen-laptop incident would look like now. Aisha explains that an encrypted laptop's data would be unreadable, likely changing the breach assessment.
Synthetic case. Synthetic.
Errors and limits
Short evaluation periods may not show seasonal problems.
Exercises
- EX115 — Lab results. Give the four before-and-after figures. Rubric: 3 = 32→45, 26→41, 4→0, 6→1; 2 = three; 1 = two; 0 = none.
- EX116 — Patching target. Was the patching target met? Rubric: 3 = 41/45 = 91% ≥ 90%; 2 = partial; 1 = vague; 0 = none.
- EX117 — Incident now. How would the incident differ now? Rubric: 3 = data unreadable; 2 = partial; 1 = vague; 0 = none.
Synthesis
Evaluate against pre-set criteria and state limits.
Continue studying — chapter 39 of 40
← Chapter 38 (draft) · Course chapter index · Chapter 40 (draft) →