OpenCyber Academy
Doctoral research · Chapter 5 · English
Course code: OCA-186Chapter 05: OCA-186-C005

Chapter 5 — Main study and independent validation

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 05 published. Map, example, EX027.

flowchart TD
  N0["Learning outcomes"]
  N1["5.1 Running the analysis"]
  N2["Lab — Difference-in-differences of median critical-p"]
  N3["5.2 Results"]
  N4["5.3 Interpreting honestly"]
  N5["5.4 Independent validation"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX027"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["5.1 Running the analysis"]
  N2["Lab — Difference-in-differences of median critical-p"]
  N3["5.2 Results"]
  N4["5.3 Interpreting honestly"]
  N5["5.4 Independent validation"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX027"]

[Learning outcomes]
    |
    v
[5.1 Running the analysis]
    |
    v
[Lab — Difference-in-differences of median critical-p]
    |
    v
[5.2 Results]
    |
    v
[5.3 Interpreting honestly]
    |
    v
[5.4 Independent validation]
    |
    v
[Worked example] --> [Exercise EX027]

Learning outcomes

You will (1) run a pre-registered difference-in-differences analysis; (2) interpret an interval that includes zero; (3) arrange independent validation of the analysis.

5.1 Running the analysis

With the pre-registration frozen, Yusuf extracts synthetic patch records for three hundred critical vulnerability instances in each of four groups: adopting departments before and after risk-ranked patching, and comparison departments over the same two periods. He computes median latency in each group, the difference-in-differences of medians and a bootstrap confidence interval by resampling each group a thousand times with a fixed random seed. The script uses only Python's statistics and random modules, so reviewers can rerun it without special software. All four groups use the same extraction rule.

Lab — Difference-in-differences of median critical-patch latency with a bootstrap interval

"""Difference-in-differences of median critical-patch latency with bootstrap interval (stdlib only)."""
import random, statistics as st
random.seed(23)
def gen(n, m): return [random.expovariate(1/m) for _ in range(n)]
t_pre, t_post = gen(300, 21), gen(300, 12)   # departments using risk-ranked patching
c_pre, c_post = gen(300, 20), gen(300, 18)   # comparison departments
did = (st.median(t_post) - st.median(t_pre)) - (st.median(c_post) - st.median(c_pre))
bs = []
for _ in range(1000):
    r = lambda x: [random.choice(x) for _ in x]
    bs.append((st.median(r(t_post)) - st.median(r(t_pre))) - (st.median(r(c_post)) - st.median(r(c_pre))))
bs.sort()
print(f"medians days: treated {st.median(t_pre):.1f} -> {st.median(t_post):.1f}; comparison {st.median(c_pre):.1f} -> {st.median(c_post):.1f}")
print(f"difference-in-differences {did:.1f} days; 95% bootstrap CI {bs[25]:.1f} to {bs[974]:.1f}")

Run with python3 lab_patch.py. Verified output:

medians days: treated 14.6 -> 10.1; comparison 12.6 -> 11.6
difference-in-differences -3.4 days; 95% bootstrap CI -7.4 to 1.2

5.2 Results

In adopting departments, median latency falls from 14.6 to 10.1 days. In comparison departments, it falls from 12.6 to 11.6 days. The difference-in-differences is minus 3.4 days, suggesting adopters improved by about three and a half days more than comparison departments. However, the 95 percent bootstrap interval runs from minus 7.4 to plus 1.2 days, so it includes zero. The data are consistent with a useful improvement, but also with no effect at all. Yusuf presents both group medians and the interval in one table, so readers see the raw changes as well as the single summary estimate. He avoids rounding the interval to hide its upper bound.

5.3 Interpreting honestly

Yusuf resists the temptation to describe the result as showing that risk-ranked patching works. Following his pre-registration, he reports the estimate, the interval and the fact that the hypothesis is not confirmed at the planned level of confidence. He also reports the pre-adoption medians, which show adopters starting slower than comparison departments, raising the possibility that part of the fall reflects catching up. Prof. Hald notes that an honest inconclusive result is a contribution, because it tells practitioners the effect is uncertain.

5.4 Independent validation

Yusuf sends his frozen script, synthetic data and pre-registration to a researcher at another university who has no stake in the result. She reruns the analysis and obtains identical figures, then tries a reasonable alternative, comparing means of log latency, and finds a similar direction and an interval that also includes zero. Her written report confirms that the conclusion does not depend on one analytic choice. Yusuf includes the report in his thesis appendix, with her permission. Her report names every file she checked.

Worked example

Yusuf runs the pre-registered analysis, finds medians falling 14.6 to 10.1 in adopters and 12.6 to 11.6 in comparisons, a difference of minus 3.4 days with an interval from minus 7.4 to 1.2, reports it as inconclusive and has it independently reproduced.

Synthetic case. Records are synthetic; outputs come from lab_patch.py.

Errors and limits

Describing an interval that includes zero as proof of an effect misleads practitioners.

Exercises

Worked answers

EX027. From 14.6 to 10.1 days.

EX028. Minus 3.4 days.

EX029. From minus 7.4 to plus 1.2 days.

EX030. The data are also consistent with no effect.

EX031. Adopters started with slower patching than comparison departments.

EX032. Comparing means of log latency.