Chapter 6 — Cryptography and trust services: controlled evaluation
Lesson map
Lesson 06 published. Map, example, EX026.
flowchart TD
N0["Learning outcomes"]
N1["6.1 Designing the evaluation"]
N2["Lab — Paired evaluation"]
N3["6.2 Reading the results"]
N4["6.3 Threats to the conclusion"]
N5["6.4 From evaluation to decision"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX026"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["6.1 Designing the evaluation"]
N2["Lab — Paired evaluation"]
N3["6.2 Reading the results"]
N4["6.3 Threats to the conclusion"]
N5["6.4 From evaluation to decision"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX026"]
[Learning outcomes]
|
v
[6.1 Designing the evaluation]
|
v
[Lab — Paired evaluation]
|
v
[6.2 Reading the results]
|
v
[6.3 Threats to the conclusion]
|
v
[6.4 From evaluation to decision]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX026]
Learning outcomes
You will (1) design a controlled evaluation of a cryptographic baseline rollout; (2) analyse paired before-and-after data; (3) report effect, uncertainty and threats.
6.1 Designing the evaluation
Before rolling the TLS baseline company-wide, Leila evaluates it at eight sites. Each site counts misconfigured endpoints, those accepting legacy protocols, weak cipher suites or expired certificates, using the same scanner profile before and three months after applying the baseline templates and automated certificate renewal. Paired measurement controls for site differences such as size and age. She registers the plan, the outcome definition and the analysis with Prof. Rao before collecting the after data, so that she cannot adjust the outcome definition to flatter the result.
Lab — Paired evaluation
"""Before/after evaluation of a TLS baseline rollout across 8 synthetic sites, with a sign-flip test."""
import random
BEFORE = [14, 9, 11, 6, 17, 8, 12, 10] # misconfigured endpoints per site
AFTER = [5, 4, 6, 3, 9, 5, 4, 6]
def main():
d = [b - a for b, a in zip(BEFORE, AFTER)]
obs = sum(d) / len(d)
rng = random.Random(11); n = 20000
hits = sum(sum(x if rng.random() < 0.5 else -x for x in d) / len(d) >= obs for _ in range(n))
print(f"mean reduction {obs:.2f} endpoints per site; every site improved: {all(x > 0 for x in d)}")
print(f"sign-flip p = {hits / n:.4f} (seed 11, {n} draws)")
if __name__ == "__main__":
main()
Run with python3 lab_eval.py. Verified output:
mean reduction 5.62 endpoints per site; every site improved: True
sign-flip p = 0.0036 (seed 11, 20000 draws)
6.2 Reading the results
Every site improves, and the mean reduction is 5.62 misconfigured endpoints per site. A sign-flip test asks how often a mean reduction this large would occur if each site's change were equally likely to be positive or negative; with seed 11 and 20,000 draws the p-value is 0.0036, close to the minimum possible for eight sites, 1/256. Leila reports the absolute counts before and after per site, not only the mean, because a manager in the largest port wants to see their own site's numbers and because skewed sites could otherwise dominate.
6.3 Threats to the conclusion
Leila lists threats. Without a control group, other changes during the three months, such as a separate patching campaign, could explain part of the improvement. The scanner sees only reachable endpoints. Staff knew they were being measured. Eight sites are few, and the pilot sites volunteered, so they may be more capable than others. She recommends a staggered rollout for remaining sites, which creates natural comparison groups, and continued measurement after the initial attention fades. She lists each threat with its likely direction of bias.
6.4 From evaluation to decision
The evaluation supports rolling out the baseline, with caveats. Leila updates the decision record from Chapter 1 with the evidence, notes the threats and adds a monitoring requirement: the misconfiguration count per site becomes a quarterly metric. She also reports costs: about two engineer-weeks per site, mostly spent on legacy clients. Including costs lets the board compare this programme with others competing for the same budget, which is the real decision context at organisation scale. The board approves the rollout for the remaining sites.
Worked example
Leila's eight-site paired evaluation shows a mean reduction of 5.62 misconfigured endpoints (sign-flip p = 0.0036), with threats and costs reported and a staggered rollout recommended.
Synthetic case. Site data are invented.
Errors and limits
Before-and-after designs without controls are vulnerable to concurrent changes. Volunteer sites may not represent the rest.
Exercises
- EX026 — Paired design. Why measure the same sites before and after? Rubric: 3 = pairing controls for site differences such as size and age;; 2 = control;; 1 = vague;; 0 = none.
- EX027 — Evaluation numbers. Report the mean reduction and p-value. Rubric: 3 = 5.62 per site, p = 0.0036;; 2 = one;; 1 = wrong;; 0 = none.
- EX028 — Minimum p. Why is 0.0036 close to the minimum possible? Rubric: 3 = eight sites allow at most 1/256 ≈ 0.0039 when all improve;; 2 = 1/256;; 1 = vague;; 0 = none.
- EX029 — Concurrent change. Which threat arises from the lack of a control group? Rubric: 3 = other changes such as a patching campaign could explain improvement;; 2 = concurrent;; 1 = vague;; 0 = none.
- EX030 — Cost reporting. Why include the two engineer-weeks per site cost? Rubric: 3 = lets the board compare against other programmes competing for budget;; 2 = budget;; 1 = irrelevant;; 0 = none.
Synthesis
Pre-registered paired evaluations, with threats and costs, turn a promising baseline into evidence for organisation-wide decisions.
Continue studying — chapter 06 of 12
← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →