Chapter 6 — Data collection and analysis
Lesson map
Lesson 06 published. Map, example, EX033.
flowchart TD
N0["Learning outcomes"]
N1["6.1 The assembled dataset"]
N2["6.2 Rates and the rate ratio"]
N3["Lab — Rate ratio with bootstrap"]
N4["6.3 Reading the interval"]
N5["6.4 Stratifying by ownership"]
N6["6.5 The change-collision description"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX033"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["6.1 The assembled dataset"]
N2["6.2 Rates and the rate ratio"]
N3["Lab — Rate ratio with bootstrap"]
N4["6.3 Reading the interval"]
N5["6.4 Stratifying by ownership"]
N6["6.5 The change-collision description"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX033"]
[Learning outcomes]
|
v
[6.1 The assembled dataset]
|
v
[6.2 Rates and the rate ratio]
|
v
[Lab — Rate ratio with bootstrap]
|
v
[6.3 Reading the interval]
|
v
[6.4 Stratifying by ownership]
|
v
[6.5 The change-collision description]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX033]
Learning outcomes
You will (1) compute incident rates per certificate-year; (2) express uncertainty for a rate ratio with a bootstrap; (3) compare overall and stratified estimates.
6.1 The assembled dataset
Full extraction covers two years. Manual certificates contribute 540 certificate-years and 23 linked expiry incidents; ACME certificates contribute 900 certificate-years and 7 incidents. Forty-one rows remain with unknown renewal mode and two incidents; Felipe reports them separately and runs a check showing that assigning them all to either group changes no conclusion. Every number traces back to a query and a coding log entry. Before running any comparison he re-reads the proposal to confirm that the analysis matches what he registered, including the seed and the resample count.
6.2 Rates and the rate ratio
A rate divides events by exposure. Manual renewal shows 4.26 incidents per 100 certificate-years and ACME 0.78, giving a rate ratio of 5.48: manual certificates failed about five and a half times as often per unit of exposure. A ratio alone hides uncertainty, and seven ACME incidents is a small number. The bootstrap estimates uncertainty by resampling certificate-years with replacement thousands of times and recomputing the ratio, then reading the middle 95 percent of the resulting values. It makes few assumptions, which suits sparse event data.
Lab — Rate ratio with bootstrap
"""Expiry incidents per 100 certificate-years with a bootstrap interval for the rate ratio (synthetic)."""
import random
MANUAL = [1] * 23 + [0] * 517 # 540 manual certificate-years, 23 incidents
ACME = [1] * 7 + [0] * 893 # 900 ACME certificate-years, 7 incidents
def rate(x): return 100 * sum(x) / len(x)
def main():
rr = rate(MANUAL) / rate(ACME)
rng = random.Random(7); boots = []
for _ in range(5000):
m = [rng.choice(MANUAL) for _ in MANUAL]; a = [rng.choice(ACME) for _ in ACME]
if sum(a): boots.append(rate(m) / rate(a))
boots.sort()
lo, hi = boots[int(0.025 * len(boots))], boots[int(0.975 * len(boots))]
print(f"manual {rate(MANUAL):.2f} per 100 cert-years, acme {rate(ACME):.2f}")
print(f"rate ratio {rr:.2f}, bootstrap 95% interval {lo:.2f}-{hi:.2f} (seed 7, 5000 resamples)")
if __name__ == "__main__":
main()
Run with python3 lab_rate.py. Verified output:
manual 4.26 per 100 cert-years, acme 0.78
rate ratio 5.48, bootstrap 95% interval 2.50-17.50 (seed 7, 5000 resamples)
6.3 Reading the interval
The bootstrap interval runs from 2.50 to 17.50 with seed 7 and 5,000 resamples. The lower bound well above 1 means the data are hard to reconcile with equal rates, but the interval is wide and asymmetric because a handful of ACME incidents dominate the denominator of the ratio. Felipe reports the interval in his abstract next to the point estimate, writing "about 2.5 to 17.5 times higher" rather than "5.48 times higher", because the single number would suggest more precision than seven events can give. A reader can then judge for themselves how much weight the finding deserves.
6.4 Stratifying by ownership
Within owner-known certificates, manual renewal shows 9 incidents in 335 certificate-years, 2.69 per 100, and ACME 2 in 711, 0.28 per 100, a ratio near 9.6. Within owner-missing certificates, manual shows 14 in 205, 6.83 per 100, and ACME 5 in 189, 2.65 per 100, a ratio near 2.6. Two lessons follow. Automation is associated with lower rates in both strata, so ownership does not explain the effect away. And owner-missing certificates fail more often whatever their mode: an orphaned ACME certificate fails about as often as an owned manual one. Ownership therefore matters on its own, and Felipe flags it as a second lever for the discussion chapter.
6.5 The change-collision description
Five of seven ACME incidents followed an infrastructure change within seven days, matching the pilot impression. Felipe reports this descriptively, as the proposal required, and adds one concrete observation from logs: in four of those five cases the ACME client had logged renewal errors for more than a week before expiry, but logs were not forwarded to monitoring. That finding is directly actionable: alerting on renewal failures, which SP 800-57 Part 1 would treat as part of planned key transition, closes a gap automation alone leaves open.
Worked example
Felipe reports 4.26 versus 0.78 incidents per 100 certificate-years, a rate ratio of 5.48 with a bootstrap interval of 2.50 to 17.50, a lower ACME rate in both ownership strata, and unmonitored renewal errors before most ACME incidents.
Synthetic note. Exposure, incidents and strata are invented teaching data.
Errors and limits
Bootstrap intervals from very few events are unstable; with seven events, report them as rough. Stratification controls ownership only.
Exercises
- EX033 — Rates. Report both renewal-mode rates per 100 certificate-years. Rubric: 3 = manual 4.26, ACME 0.78;; 2 = one;; 1 = counts;; 0 = none.
- EX034 — Bootstrap interval. Give the rate ratio and its interval with the seed. Rubric: 3 = 5.48, 2.50–17.50, seed 7, 5000 resamples;; 2 = ratio and interval;; 1 = ratio only;; 0 = none.
- EX035 — Wide interval. Why is the interval wide and asymmetric? Rubric: 3 = only seven ACME incidents dominate the ratio's denominator;; 2 = small sample;; 1 = bad code;; 0 = none.
- EX036 — Stratum ratios. Give the rate ratio within each ownership stratum. Rubric: 3 = about 9.6 owner-known, about 2.6 owner-missing;; 2 = one;; 1 = rates only;; 0 = none.
- EX037 — Orphan comparison. How does an orphaned ACME certificate compare with an owned manual one? Rubric: 3 = roughly the same rate (2.65 vs 2.69 per 100);; 2 = similar;; 1 = much lower;; 0 = none.
- EX038 — Log finding. What did logs show before ACME incidents, and what fix follows? Rubric: 3 = renewal errors logged over a week in 4 of 5 but not monitored; alert on renewal failures;; 2 = finding only;; 1 = fix only;; 0 = none.
Synthesis
Rates, intervals and strata together show a robust association between automation and fewer expiries, and expose ownership and monitoring as independent levers.
Continue studying — chapter 06 of 08
← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →