Chapter 9 — Evidence and measurement
Lesson map
Lesson 09 published. Map, example, EX025.
flowchart TD
N0["Learning outcomes"]
N1["9.1 Freshness"]
N2["Lab — version age"]
N3["9.2 Lab result"]
N4["9.3 Remediation time"]
N5["9.4 Behaviour"]
N6["9.5 Automatic evidence"]
N7["9.6 Presenting"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX025"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["9.1 Freshness"]
N2["Lab — version age"]
N3["9.2 Lab result"]
N4["9.3 Remediation time"]
N5["9.4 Behaviour"]
N6["9.5 Automatic evidence"]
N7["9.6 Presenting"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX025"]
[Learning outcomes]
|
v
[9.1 Freshness]
|
v
[Lab — version age]
|
v
[9.2 Lab result]
|
v
[9.3 Remediation time]
|
v
[9.4 Behaviour]
|
v
[9.5 Automatic evidence]
|
v
[9.6 Presenting]
|
v
[Worked example] --> [Exercise EX025]
Learning outcomes
You will (1) measure dependency freshness and remediation; (2) choose measures that drive the right behaviour; (3) collect evidence automatically; (4) present trends.
9.1 Freshness
How old is each installed library version? Old versions miss fixes and are more likely to be affected by advisories. Kula et al. (2018) found developers often leave dependencies outdated. The lab measures this for Greenleaf on 1 March 2026. upload-kit's installed version was released on 2 June 2024, 637 days earlier, which places it among the oldest libraries in the booking site. Age is easy to measure from the inventory and release dates, which makes it a practical first signal for a small agency. Diego treats it as a prompt to look closer, not as proof that a library is unsafe, because some old versions have no known flaws. The 365-day threshold the team agreed marks where that closer look becomes routine.
Lab — version age
"""Days since each dependency's installed version was released, flagging versions older than 365 days (synthetic)."""
from datetime import date
today = date(2026, 3, 1)
deps = {"upload-kit": date(2024, 6, 2), "forms": date(2025, 11, 20), "date-utils": date(2023, 1, 15), "mailer": date(2026, 2, 3)}
for name, released in sorted(deps.items(), key=lambda d: d[1]):
age = (today - released).days
print(f"{name:11} {age:4} days {'STALE' if age > 365 else ''}")
Run with python3 lab_age.py. Verified output:
date-utils 1141 days STALE
upload-kit 637 days STALE
forms 101 days
mailer 26 days
9.2 Lab result
For Greenleaf, date-utils is 1,141 days old and upload-kit 637 days; both are flagged stale (over 365 days). forms is 101 days old and mailer 26 days. upload-kit's age explains why the January advisory found it vulnerable. The lab calculates each age by subtracting the release date from 1 March 2026, so date-utils, released on 15 January 2023, is the oldest by a wide margin. The stale flag uses a strict comparison, so an age of exactly 365 days would not be marked stale. Under the stricter 180-day threshold Diego uses for upload, authentication and payment libraries, upload-kit's 637 days would be flagged even more clearly. forms at 101 days and mailer at 26 days pass either threshold. The lab hard-codes the release dates, so Diego checks each against the registry before trusting the result.
9.3 Remediation time
Days from an advisory's publication to the fix being deployed, per severity. For Brightwave the clock starts when an advisory is published and stops when the fixed version is deployed to a client, not when the pull request is merged. Measured this way, January's upload-kit case shows the gap clearly: the Greenleaf fix took a day, but finding museum-tickets took three more. Reporting the measure per severity lets the team check it against the target of seven days for high advisories.
9.4 Behaviour
Measures can drive the wrong behaviour: counting "libraries updated" could encourage risky bulk updates without tests. Brightwave measures outcomes: stale count and remediation time. Stale count rewards keeping libraries current, and remediation time rewards fixing what advisories actually flag. Neither rewards touching many libraries at once without tests, which is the behaviour a raw update count would encourage. Before choosing measures, the team asks what a developer would do to improve each number quickly. If the quick route is unsafe, the measure is rejected or paired with one that would catch it.
9.5 Automatic evidence
Measures come from the nightly inventory and deployment records, not manual reports. Because the figures come from the same nightly inventory and deployment records used for daily work, nobody has to compile them by hand, and nobody can make a month look better by choosing what to report. If the inventory misses a project, the measures miss it too, which is why the missing-project count matters later in the course. The records also carry dates, so each figure can be traced back to its source.
9.6 Presenting
A monthly chart shows stale libraries and remediation times across clients. The chart covers all 23 client projects and shows stale counts falling from 64 towards 18 over six months, alongside remediation times by severity. Aiko uses it in the monthly team meeting to show which projects need attention, and the two legacy projects stand out because most of the remaining stale versions sit there. Each chart shows the threshold and the date of the data, so readers can compare months fairly. Showing both figures together stops one improving while the other quietly worsens.
Choosing thresholds
The 365-day stale threshold is a starting point that the team agreed together. For libraries handling uploads, authentication or payments, Diego uses a stricter threshold of 180 days, because flaws there matter more. Thresholds are reviewed each quarter against what the advisories actually show: if many advisories hit versions younger than the threshold, it is too loose. The lab's docstring states the 365-day threshold it applies, so anyone reading the code knows which rule produced the STALE flags.
Reading the lab
The lab compares each installed version's release date with today's date and prints the age in days and a status. Diego checks that the dates are correct before trusting the result.
Worked example
After six months of weekly updates, the number of stale library versions across all clients falls from 64 to 18. The remaining 18 are mostly in two legacy projects scheduled for rebuild.
Synthetic case. Ages and counts are synthetic.
Errors and limits
Age is a rough proxy; a new version can still have flaws, an old one may be fine.
Exercises
- EX025 — Lab stale. Which libraries were stale? Rubric: 3 = date-utils 1,141; upload-kit 637; 2 = one; 1 = vague; 0 = none.
- EX026 — Bad measure. Why not count libraries updated? Rubric: 3 = encourages risky bulk updates; 2 = partial; 1 = vague; 0 = none.
- EX027 — Six-month result. What did six months of updates achieve? Rubric: 3 = stale 64 to 18; 2 = partial; 1 = vague; 0 = none.
Synthesis
Measure freshness and remediation from automatic evidence.
Continue studying — chapter 09 of 30
← Chapter 08 (draft) · Course chapter index · Chapter 10 (draft) →