Chapter 39 — Capstone evaluation
Lesson map
Lesson 39 published. Map, example, EX115.
flowchart TD
N0["Learning outcomes"]
N1["39.1 Collecting results"]
N2["Lab — Before and after"]
N3["39.2 Reading the results"]
N4["39.3 Alternative explanations"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX115"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["39.1 Collecting results"]
N2["Lab — Before and after"]
N3["39.2 Reading the results"]
N4["39.3 Alternative explanations"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX115"]
[Learning outcomes]
|
v
[39.1 Collecting results]
|
v
[Lab — Before and after]
|
v
[39.2 Reading the results]
|
v
[39.3 Alternative explanations]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX115]
Learning outcomes
You will (1) compare before and after measures; (2) judge each against its target; (3) discuss alternative explanations.
39.1 Collecting results
After three months, Elena collects the after-period values using exactly the same definitions and queries as the before period. She checks each against its source and records the export dates. Where data is missing, she follows her plan. She also gathers her log of other events during the period, such as two new starters and one system update, so she can discuss them when interpreting results. She stores the after-period exports alongside the before-period files in the same folder structure, and records the script version used for each calculation, so anyone can repeat the comparison exactly.
Lab — Before and after
"""Before/after evaluation of synthetic access-hygiene measures for the capstone (stdlib only)."""
m = [("MFA coverage %", 77.1, 97.9, "up"), ("Leavers disabled within 1 day %", 50.0, 100.0, "up"),
("Shared accounts", 2, 0, "down"), ("Admin accounts", 9, 4, "down"), ("Unused licences", 23, 6, "down")]
for name, b, a, d in m:
ok = (a > b) if d == "up" else (a < b)
print(f"{name:34} before={b:<6} after={a:<6} {'improved' if ok else 'not improved'}")
Run with python3 lab_eval.py. Verified output:
MFA coverage % before=77.1 after=97.9 improved
Leavers disabled within 1 day % before=50.0 after=100.0 improved
Shared accounts before=2 after=0 improved
Admin accounts before=9 after=4 improved
Unused licences before=23 after=6 improved
39.2 Reading the results
All five measures improved. Multi-factor coverage rose from 77.1 to 97.9 percent, above the 95 percent target, with about two points from clean-up. Leavers disabled within one day rose from 50 to 100 percent over the after period. Both shared accounts were removed. Administrator accounts fell from nine to four. Unused paid licences fell from 23 to 6, saving money that helps justify the work. Elena reports these with counts and notes that small numbers, especially for leavers, mean results should be confirmed over a longer period.
39.3 Alternative explanations
Could anything else explain the improvements? The leaver result could partly reflect fewer leavers in the after period, so she reports counts: nine before and seven after. Coverage gains could reflect clean-up, which she has separated. The new IT staff member may have helped with administrator reduction, which she acknowledges. None of these explains the overall pattern, which matches the timing of each change. She concludes the changes very likely caused most of the improvement, with stated uncertainty. Her supervisor agrees with this reading.
Worked example
All five measures improve, including coverage from 77.1 to 97.9 percent and leavers within a day from 50 to 100 percent; alternatives are discussed with counts.
Synthetic case. Measures, counts and events are synthetic.
Errors and limits
Claiming every improvement was caused by the project without discussing alternatives overstates the evidence.
Exercises
- EX115 — Coverage change. How did multi-factor coverage change in the lab? Rubric: 3 = 77.1% to 97.9%, above target;; 2 = rose;; 1 = fell;; 0 = none.
- EX116 — Licence saving. How did unused licences change? Rubric: 3 = 23 to 6;; 2 = fell;; 1 = rose;; 0 = none.
- EX117 — Leaver counts. How many leavers were in each period? Rubric: 3 = nine before, seven after;; 2 = counts;; 1 = vague;; 0 = none.
Synthesis
Consistent measurement, targets and honest discussion of alternatives make the capstone evaluation credible.
Continue studying — chapter 39 of 40
← Chapter 38 (draft) · Course chapter index · Chapter 40 (draft) →