Chapter 5 — Main study and independent validation
Lesson map
Lesson 05 published. Map, example, EX027.
flowchart TD
N0["Learning outcomes"]
N1["5.1 Design"]
N2["5.2 Exposure outcome"]
N3["Lab — exposure window"]
N4["5.3 Lab result"]
N5["5.4 Cost"]
N6["5.5 Independent validation"]
N7["Threats to validity"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX027"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["5.1 Design"]
N2["5.2 Exposure outcome"]
N3["Lab — exposure window"]
N4["5.3 Lab result"]
N5["5.4 Cost"]
N6["5.5 Independent validation"]
N7["Threats to validity"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX027"]
[Learning outcomes]
|
v
[5.1 Design]
|
v
[5.2 Exposure outcome]
|
v
[Lab — exposure window]
|
v
[5.3 Lab result]
|
v
[5.4 Cost]
|
v
[5.5 Independent validation]
|
v
[Threats to validity]
|
v
[Worked example] --> [Exercise EX027]
Learning outcomes
You will (1) run the main comparison; (2) estimate the effect with an interval; (3) measure cost; (4) have results validated independently.
5.1 Design
Eight production clusters move from daily scanning to admission enforcement in four steps of two clusters, one step a month. Each cluster contributes exposure measurements before and after its switch.
The design gives each step a comparison: in the first month, two clusters switch and six still scan; by the fourth month all eight have switched. Because every cluster eventually receives admission enforcement, no team is denied the control for research purposes, and each cluster serves as its own comparison before and after.
5.2 Exposure outcome
For each risky workload configuration, exposure is hours from becoming live to removal (scanning) or to an approved exception (admission, where a blocked deployment has zero exposure unless an exception is granted).
The definition treats a blocked deployment as zero exposure because the risky configuration never ran. That choice was later challenged in critique, because a team might run the same workload elsewhere, and Amara's sensitivity analysis tests it. Exceptions count as exposure from the time they are granted, which is why exception expiry matters.
Lab — exposure window
"""Exposure window of risky configs: daily scan vs admission check, with a bootstrap interval (stdlib only)."""
import random, statistics
random.seed(7)
daily = [3, 9, 14, 18, 21, 22, 23, 11, 6, 17, 20, 13, 8, 19, 24, 5, 16, 12, 10, 15] # hours exposed
admission = [0, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0] # hours exposed (exceptions)
d = statistics.median(daily) - statistics.median(admission)
boots = []
for _ in range(2000):
s1 = [random.choice(daily) for _ in daily]
s2 = [random.choice(admission) for _ in admission]
boots.append(statistics.median(s1) - statistics.median(s2))
boots.sort()
print(f"median daily {statistics.median(daily)} h, admission {statistics.median(admission)} h, difference {d}")
print(f"95% bootstrap interval {boots[49]}-{boots[1949]} h (seed 7, 2000 resamples)")
Run with python3 lab_window.py. Verified output:
median daily 14.5 h, admission 0.0 h, difference 14.5
95% bootstrap interval 10.5-18.5 h (seed 7, 2000 resamples)
5.3 Lab result
On the synthetic sample of 20 configurations per arm, the median exposure under daily scanning is 14.5 hours and under admission 0.0 hours, a difference of 14.5 hours. The 95% bootstrap interval is 10.5 to 18.5 hours (seed 7, 2000 resamples), well clear of the no-effect rule.
Under scanning, exposure depends on when a configuration goes live relative to the daily scan and how quickly a team fixes it after being told. Under admission, most risky configurations never run. Reporting the bootstrap interval with its seed and number of resamples means anyone with the synthetic data can reproduce the interval exactly.
5.4 Cost
Deployment rejections fall from 7% in the first fortnight to 3% after, below the 5% ceiling; exception requests total 11, of which 9 are approved with reasons.
The fall from 7% to 3% reflects teams learning the rules in the first fortnight, which is why the ceiling applies after it. The 11 exception requests are the second cost: each takes a reviewer's time and leaves a risky configuration live. In the step-two clusters, exceptions approved without end dates raised exposure, and Meridian added 30-day expiry. The 2 refused requests are recorded with their reasons, so the panel's condition that exception workloads be reported separately covers refusals as well as approvals.
5.5 Independent validation
A researcher outside the project reruns the scripts on the archived synthetic and de-identified data and reproduces the medians and interval exactly.
Exact reproduction is possible because the bootstrap uses a recorded seed and number of resamples. It also tests that the archived data are complete: any missing file or undocumented step would have shown up as a different median or interval. The validator works only from the archive and its README, with no help from Amara, so any gap in the instructions would also have shown up as a failed rerun. The validator's report is filed in the archive.
Threats to validity
Amara lists the main threats. Teams knew the switch dates, so they may have tidied configurations beforehand; she checks the month before each switch and finds no drop in new risky configurations. Some workloads moved between clusters, which she handles by following workload identity. The study covers one organisation, so external validity is limited; the method, not the size of the effect, is what she claims transfers. Writing these threats down alongside the result lets readers judge it fairly.
Worked example
In the step-two clusters, exposure is higher than expected because exceptions were approved without end dates. Amara reports this separately, as the panel required, and Meridian adds 30-day expiry to exceptions.
Synthetic case. All figures are synthetic.
Errors and limits
A stepped design reduces but does not remove time effects; report calendar-time checks.
Exercises
- EX027 — Design. Describe the rollout design. Rubric: 3 = 8 clusters, 4 steps of 2, monthly; 2 = partial; 1 = vague; 0 = none.
- EX028 — Lab medians. What were the medians and difference? Rubric: 3 = 14.5 vs 0.0; 14.5; 2 = partial; 1 = vague; 0 = none.
- EX029 — Interval. What was the interval? Rubric: 3 = 10.5-18.5 h; 2 = partial; 1 = vague; 0 = none.
- EX030 — Cost result. Was the cost ceiling met? Rubric: 3 = yes, 3% after fortnight; 2 = partial; 1 = vague; 0 = none.
- EX031 — Validation. How were results validated? Rubric: 3 = outside researcher reran scripts; 2 = partial; 1 = vague; 0 = none.
- EX032 — Exceptions. What raised step-two exposure? Rubric: 3 = exceptions without end dates; 2 = partial; 1 = vague; 0 = none.
Synthesis
Estimate with intervals, measure costs and invite independent reruns.
Continue studying — chapter 05 of 08
← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →