Chapter 15 — Evidence and measurement
Lesson map
Lesson 15 published. Map, example, EX043.
flowchart TD
N0["Learning outcomes"]
N1["15.1 Choosing measures"]
N2["15.2 Skewed data"]
N3["Lab — Days to patch: median and share within target"]
N4["15.3 Honest presentation"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX043"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["15.1 Choosing measures"]
N2["15.2 Skewed data"]
N3["Lab — Days to patch: median and share within target"]
N4["15.3 Honest presentation"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX043"]
[Learning outcomes]
|
v
[15.1 Choosing measures]
|
v
[15.2 Skewed data]
|
v
[Lab — Days to patch: median and share within target]
|
v
[15.3 Honest presentation]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX043]
Learning outcomes
You will (1) choose measures that reflect control performance; (2) use medians and thresholds for skewed data; (3) present measures honestly.
15.1 Choosing measures
Elmfield wants to know whether its update process works. Sofia chooses two measures: the median number of days to apply critical updates and the share applied within the thirty-day target. She learns that measures should relate to the control's purpose, which is to shorten the time vulnerabilities remain open. Counting updates installed would be an activity measure; time to install is a performance measure closer to what matters for risk. Ten of fifteen updates within thirty days is a performance figure; fifteen updates installed would only show activity.
15.2 Skewed data
The lab examines fifteen critical updates. The median is twenty-one days and ten of fifteen, sixty-seven percent, were applied within thirty days. The three slowest took forty-eight, sixty-two and ninety days. Sofia notes that the mean would be pulled up by these long delays, so the median better describes a typical update while the slowest cases deserve separate attention. Most slow updates affected the file server, which staff avoid restarting during working hours. This file server pattern links the skewed figures to a cause the maintenance window can address.
Lab — Days to patch: median and share within target
"""Days to patch for 15 synthetic critical updates: median and share within 30 days."""
from statistics import median
days = [3, 5, 7, 8, 12, 14, 15, 21, 25, 29, 33, 41, 48, 62, 90]
print(f"updates {len(days)}; median {median(days)} days")
w = sum(d <= 30 for d in days); print(f"within 30 days {w}/{len(days)} = {w/len(days):.0%}")
print(f"slowest three {days[-3:]}")
Run with python3 lab_patch.py. Verified output:
updates 15; median 21 days
within 30 days 10/15 = 67%
slowest three [48, 62, 90]
15.3 Honest presentation
Sofia presents both measures with the list of slow updates and their cause. She avoids saying that most updates are on time without adding that a third are not. She recommends a monthly maintenance window for the file server. Dr. Haddad notes that NIST SP 800-55v1 discusses selecting and using measures for information security, and that good measures prompt decisions rather than decorate reports. Sofia's measure leads directly to the maintenance window decision. Showing the slow updates by name lets the IT lead act on the file server rather than on Elmfield's laptops in general.
Worked example
Sofia measures a median of twenty-one days with sixty-seven percent on target and traces slow updates to the file server.
Synthetic case. Update data are synthetic; NIST SP 800-55v1 is real.
Errors and limits
Small samples vary. Medians hide the slowest cases.
Exercises
- EX043 — Performance measure. Why is time to install a better measure than updates installed? Rubric: 3 = it relates to how long vulnerabilities stay open;; 2 = performance;; 1 = activity;; 0 = none.
- EX044 — Patch results. What median and on-target share did the lab report? Rubric: 3 = 21 days, 67%;; 2 = median 21;; 1 = vague;; 0 = none.
- EX045 — Slow cause. Why were the slowest updates late? Rubric: 3 = the file server was not restarted during working hours;; 2 = server;; 1 = vague;; 0 = none.
Synthesis
Good measures reflect purpose, handle skew and lead to decisions.
Continue studying — chapter 15 of 40
← Chapter 14 (draft) · Course chapter index · Chapter 16 (draft) →