Chapter 5 — Main study and independent validation
Lesson map
Lesson 05 published. Map, example, EX027.
flowchart TD
N0["Learning outcomes"]
N1["5.1 Running the study"]
N2["Lab — Before and after"]
N3["5.2 Reading the result"]
N4["5.3 Independent validation"]
N5["5.4 Effect in context"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX027"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["5.1 Running the study"]
N2["Lab — Before and after"]
N3["5.2 Reading the result"]
N4["5.3 Independent validation"]
N5["5.4 Effect in context"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX027"]
[Learning outcomes]
|
v
[5.1 Running the study]
|
v
[Lab — Before and after]
|
v
[5.2 Reading the result]
|
v
[5.3 Independent validation]
|
v
[5.4 Effect in context]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX027]
Learning outcomes
You will (1) run the intervention study; (2) compare before and after; (3) validate coding independently.
5.1 Running the study
For one quarter, eight teams follow the review procedure while four continue their usual practice. Ines does not change any rules herself; teams make their own changes through their normal change process. She collects configurations at the start and end of the quarter, along with change logs and time records, and both coders recode all twelve firewalls at the end.
Running the study follows the written protocol step by step. Ines logs every deviation, such as a delayed review session or a missing configuration snapshot, with its date and reason. Deviations are reported in the thesis rather than hidden, because examiners judge validity partly on how honestly departures from the plan are described.
Lab — Before and after
"""Compare findings per 100 rules before and after a structured review, with a simple permutation test (stdlib only)."""
import random
pre = [6.1, 5.4, 7.2, 4.9, 6.8, 5.7, 6.3, 7.0]
post = [3.9, 4.1, 5.2, 3.3, 4.8, 4.0, 4.4, 5.1]
diff = sum(pre) / 8 - sum(post) / 8
random.seed(7); pool = pre + post; c = 0
for _ in range(10000):
random.shuffle(pool)
if sum(pool[:8]) / 8 - sum(pool[8:]) / 8 >= diff: c += 1
print(f"mean pre {sum(pre)/8:.2f} post {sum(post)/8:.2f} per 100 rules; difference {diff:.2f}; one-sided p ~ {c/10000:.4f}")
Run with python3 lab_trial.py. Verified output:
mean pre 6.17 post 4.35 per 100 rules; difference 1.83; one-sided p ~ 0.0006
5.2 Reading the result
In the synthetic data, errors per hundred rules on reviewed firewalls fall from a mean of 6.17 to 4.35. A permutation test, which asks how often a difference this large would appear if the labels were shuffled at random, gives a one-sided p of about 0.0006. The comparison firewalls show a small fall over the same quarter, which Ines reports alongside, since part of the change may come from general attention to rule hygiene rather than the procedure itself.
She reports the interval alongside the estimate.
5.3 Independent validation
A second coder, blind to which firewalls were reviewed, recodes a random quarter of rules. Agreement remains high, and the blinded coder's error rates show the same direction of change. Ines also asks an engineer from a fourth organisation, not involved in the study, to apply the procedure to a synthetic rule base; the engineer finds similar categories of issue, suggesting the procedure is usable by outsiders. Both checks are reported in full, including where the blinded coder disagreed.
Disagreements are documented, not quietly resolved.
5.4 Effect in context
Ines translates the result into operational terms: for a firewall of five hundred rules, the reduction corresponds to about nine fewer problematic rules after one quarter. She reports the review time per hundred rules, so readers can weigh benefit against cost, and avoids claiming any reduction in incidents, which the study did not measure.
An effect size only means something in context. Ines compares the reduction in error rate with the effort the review required, measured in engineer hours, and with variation between partners. A modest average reduction that is consistent across partners may be more useful in practice than a large effect seen in only one organisation.
Worked example
The final table lists mean errors per hundred rules, the difference, the permutation p value, the comparison group's change and review hours, with the blinded validation summarised beneath.
Synthetic case. Data are synthetic.
Errors and limits
A small p value says the difference is unlikely under random labelling; it does not measure how important the difference is.
Exercises
- EX027 — Permutation logic. Explain the permutation test in plain words. Rubric: 3 = shuffle labels, compare differences;; 2 = shuffle;; 1 = vague;; 0 = none.
- EX028 — Comparison group. Why report the comparison group's change? Rubric: 3 = separate general attention from procedure effect;; 2 = context;; 1 = vague;; 0 = none.
- EX029 — Operational translation. Convert the difference into errors avoided for 500 rules. Rubric: 3 = about nine, shown;; 2 = roughly;; 1 = vague;; 0 = none.
- EX030 — Blinded recoding. What did blinded recoding confirm? Rubric: 3 = same direction, high agreement;; 2 = agreement;; 1 = vague;; 0 = none.
- EX031 — Outsider test. Why ask an outside engineer to apply the procedure? Rubric: 3 = usability beyond partners;; 2 = generalise;; 1 = vague;; 0 = none.
- EX032 — Claim boundary. What must Ines not claim from these results? Rubric: 3 = reduction in incidents;; 2 = causation broadly;; 1 = vague;; 0 = none.
Synthesis
The main study supports H2 within its limits, and independent validation strengthens confidence in the coding.
Continue studying — chapter 05 of 08
← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →