Chapter 29 — Application and software supply chain security: measurement
Lesson map
Lesson 29 published. Map, example, EX085.
flowchart TD
N0["Learning outcomes"]
N1["29.1 What to measure"]
N2["Lab — Lab — remediation times"]
N3["29.2 Lab result"]
N4["29.3 Late items"]
N5["29.4 Perverse incentives"]
N6["29.5 Freshness"]
N7["Reading the dashboard"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX085"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["29.1 What to measure"]
N2["Lab — Lab — remediation times"]
N3["29.2 Lab result"]
N4["29.3 Late items"]
N5["29.4 Perverse incentives"]
N6["29.5 Freshness"]
N7["Reading the dashboard"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX085"]
[Learning outcomes]
|
v
[29.1 What to measure]
|
v
[Lab — Lab — remediation times]
|
v
[29.2 Lab result]
|
v
[29.3 Late items]
|
v
[29.4 Perverse incentives]
|
v
[29.5 Freshness]
|
v
[Reading the dashboard]
|
v
[Worked example] --> [Exercise EX085]
Learning outcomes
You will (1) define security metrics for application and supply chain; (2) compute remediation times against targets; (3) identify late items and causes; (4) report metrics without perverse incentives.
29.1 What to measure
Coastline measures time to remediate findings by severity, share of builds with valid provenance, number of active exceptions, and dependency freshness (how many versions behind the latest).
Each metric answers a different question. Remediation time shows how quickly known problems are fixed. Provenance coverage shows how much of what runs can be traced to reviewed source. The exception count shows how often controls are being bypassed. Freshness predicts future pain: a dependency far behind will need a large, risky upgrade when a flaw finally forces one. Metrics are chosen because someone will act on them; a number nobody can influence is dropped from the dashboard.
Lab — Lab — remediation times
"""Median days to remediate synthetic findings by severity against targets."""
from statistics import median
targets = {"critical": 7, "high": 30, "medium": 90}
found = {"critical": [2, 5, 9], "high": [12, 20, 41, 18, 33], "medium": [60, 75, 120, 44]}
for sev, days in found.items():
late = sum(d > targets[sev] for d in days)
print(f"{sev:8} n={len(days)} median={median(days)} target={targets[sev]} late={late}")
Run with python3 lab_fixtime.py. Verified output:
critical n=3 median=5 target=7 late=1
high n=5 median=20 target=30 late=2
medium n=4 median=67.5 target=90 late=1
29.2 Lab result
Critical: three findings, median 5 days, one late (9 days against a 7-day target). High: five findings, median 20 days, two late (41 and 33 against 30). Medium: four findings, median 67.5 days, one late (120 against 90).
The medians all sit inside their targets, yet four of twelve findings were late. A report showing only medians would suggest everything is fine. Counting late items separately reveals the cases that need attention. With only three critical findings, one extra late item would move the median sharply, which is the instability the errors note warns about. For an even count, as with the four medium findings, the median is the average of the middle two values, 60 and 75, which gives 67.5.
29.3 Late items
The late critical item waited for a vendor patch; the late highs needed a framework upgrade that took two sprints. Measuring lateness leads to the cause, not blame.
The two causes call for different actions. A vendor delay may be handled by a temporary control, such as isolating the component or disabling the affected feature, while waiting. A slow framework upgrade points to too much deferred maintenance; regular small upgrades would make the next one easier. Asking 'what made this slow?' instead of 'who was slow?' gets honest answers, because people are not afraid to describe the delay.
29.4 Perverse incentives
If teams are rewarded only for closing fast, they may downgrade severities. Coastline reviews severity changes monthly.
A finding downgraded from high to medium gains sixty days under the targets, which makes the numbers look better without making riders any safer. The monthly review checks each change against its evidence; changes need a written reason and a second person's agreement. Coastline also avoids rewarding individuals on remediation numbers, since that invites gaming. Teams are praised for honest reporting, including when the numbers look bad, because hidden problems cannot be fixed.
29.5 Freshness
The median dependency is one minor version behind; two are more than two major versions behind and planned for replacement.
Being one minor version behind usually means the team is current enough to take a security fix quickly, often with no code changes. Being two major versions behind means a fix may only be released for newer versions, forcing a large upgrade under pressure. Replacement is planned for those two while there is time, rather than waiting for an advisory to force the work. The freshness figure is reviewed monthly alongside remediation time, since the two tend to move together.
Reading the dashboard
The dashboard shows each metric as a trend over six months with targets as lines. Items behind target are listed by name with owners. Ingrid reviews it weekly with the team, asking about causes rather than people; the review takes 15 minutes and ends with at most three actions. Kwame finds that showing named items, not just numbers, gets them fixed faster because owners see their work.
Sharing metrics
Monthly summaries go to Coastline's leadership with a short explanation of trends.
Worked example
Kwame's dashboard shows provenance on 100% of server builds and 80% of mobile builds (the remaining 20% are older releases built before the CI move). Ingrid sets a goal to retire those releases.
Synthetic case. All measurements are synthetic.
Errors and limits
Small counts make medians unstable; show the individual items too.
Exercises
- EX085 — Lab late items. How many items were late in each severity? Rubric: 3 = critical 1, high 2, medium 1; 2 = two; 1 = one; 0 = none.
- EX086 — Critical cause. Why was the critical item late? Rubric: 3 = waited for vendor patch; 2 = partial; 1 = vague; 0 = none.
- EX087 — Incentive control. How are perverse incentives controlled? Rubric: 3 = monthly review of severity changes; 2 = partial; 1 = vague; 0 = none.
Synthesis
Measure remediation, provenance and freshness, find causes and guard against gaming.
Continue studying — chapter 29 of 50
← Chapter 28 (draft) · Course chapter index · Chapter 30 (draft) →