Chapter 4 — Baseline replication and pilot
Lesson map
Lesson 04 published. Map, example, EX021.
flowchart TD
N0["Learning outcomes"]
N1["4.1 Coding the baseline"]
N2["Lab — Size and errors"]
N3["4.2 Reading the replication"]
N4["Lab — Shadowed rules"]
N5["4.3 Piloting the procedure"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX021"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["4.1 Coding the baseline"]
N2["Lab — Size and errors"]
N3["4.2 Reading the replication"]
N4["Lab — Shadowed rules"]
N5["4.3 Piloting the procedure"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX021"]
[Learning outcomes]
|
v
[4.1 Coding the baseline]
|
v
[Lab — Size and errors]
|
v
[4.2 Reading the replication]
|
v
[Lab — Shadowed rules]
|
v
[4.3 Piloting the procedure]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX021]
Learning outcomes
You will (1) replicate the complexity-error relationship; (2) detect shadowed rules; (3) pilot the review procedure.
4.1 Coding the baseline
With the codebook fixed, two coders label every rule on the twelve firewalls, with an analysis tool assisting shadowing checks. Disagreements are resolved by discussion and recorded. Ines then counts errors per firewall and compares with rule-base size. To keep the analysis honest, she computes the correlation using ranks, which is robust to a few very large firewalls dominating the result.
Coding the baseline means classifying every rule in each partner's rule base using the codebook, before any review takes place. Ines records each decision with the rule identifier and a short reason, so disagreements can be traced. A second coder independently codes a random sample, and agreement is measured before the full baseline is accepted.
Lab — Size and errors
"""Spearman correlation between rule-base size and error count across 12 synthetic firewalls (stdlib only)."""
size = [40, 85, 120, 150, 210, 260, 330, 400, 520, 610, 780, 950]
errs = [2, 3, 3, 5, 4, 7, 6, 9, 8, 11, 12, 14]
def rank(v): s = sorted(v); return [s.index(x) + (s.count(x) - 1) / 2 + 1 for x in v]
rs, re_ = rank(size), rank(errs); n = len(size)
d2 = sum((a - b) ** 2 for a, b in zip(rs, re_))
print(f"firewalls {n}; spearman rho {1 - 6 * d2 / (n * (n * n - 1)):.3f}")
Run with python3 lab_corr.py. Verified output:
firewalls 12; spearman rho 0.977
4.2 Reading the replication
The synthetic data give a Spearman correlation of 0.977 between rule count and error count, a very strong positive relationship consistent with H1 and with Wool's earlier findings. Ines is cautious: twelve firewalls is a small sample, and larger firewalls may also be older, have more owners or face more change. Correlation does not show that size causes errors. She reports the result as a replication of an association and discusses plausible mechanisms, such as the difficulty of keeping track of how hundreds of rules interact.
Lab — Shadowed rules
"""Find shadowed rules in a synthetic first-match rule list: a later rule fully covered by an earlier one (stdlib only)."""
import ipaddress as ip
rules = [ # (id, src, dst, port, action)
(1, "10.0.0.0/8", "192.168.10.0/24", 443, "allow"), (2, "10.1.0.0/16", "192.168.10.5/32", 443, "deny"),
(3, "0.0.0.0/0", "192.168.20.0/24", 22, "deny"), (4, "10.2.0.0/16", "192.168.20.7/32", 22, "allow"),
(5, "10.3.0.0/16", "192.168.30.0/24", 3389, "allow"), (6, "10.3.4.0/24", "192.168.30.9/32", 3389, "allow")]
def covers(e, l):
return ip.ip_network(l[1]).subnet_of(ip.ip_network(e[1])) and ip.ip_network(l[2]).subnet_of(ip.ip_network(e[2])) and e[3] == l[3]
for i, l in enumerate(rules):
for e in rules[:i]:
if covers(e, l):
kind = "conflict" if e[4] != l[4] else "redundant"
print(f"rule {l[0]} shadowed by rule {e[0]} ({kind})"); break
Run with python3 lab_shadow.py. Verified output:
rule 2 shadowed by rule 1 (conflict)
rule 4 shadowed by rule 3 (conflict)
rule 6 shadowed by rule 5 (redundant)
4.3 Piloting the procedure
Before the main study, one team pilots the review on a test copy of a small rule base. The analysis lab shows how shadowing is found: rule 2 can never match because rule 1 already covers its traffic with the opposite action, a conflict that hides the engineer's intent; rule 6 is redundant under rule 5. The pilot takes the team three hours for sixty rules, longer than hoped, mainly because owners were hard to identify. Ines adds an owner-lookup step using the change log, which halves the time in a second pilot.
Worked example
The baseline shows a strong rank correlation, the shadowing analysis identifies two conflicts and one redundancy in the pilot set, and the revised procedure takes ninety minutes per sixty rules.
Synthetic case. Data and rules are synthetic.
Errors and limits
A strong correlation in a small sample can be driven by confounders such as age; report it as association.
Exercises
- EX021 — Rank correlation. Why use Spearman rather than Pearson here? Rubric: 3 = robust to outliers and non-linear monotonic relation;; 2 = robust;; 1 = vague;; 0 = none.
- EX022 — Association claim. Write a careful sentence reporting rho 0.977. Rubric: 3 = association, sample size, no causation;; 2 = association;; 1 = causal;; 0 = none.
- EX023 — Conflict versus redundancy. Distinguish a conflicting from a redundant shadowed rule. Rubric: 3 = opposite vs same action, consequences;; 2 = definitions;; 1 = vague;; 0 = none.
- EX024 — Confounders. Name two confounders for the size-error link. Rubric: 3 = age and number of owners or change rate;; 2 = one;; 1 = vague;; 0 = none.
- EX025 — Pilot finding. What did the pilot reveal and how was it fixed? Rubric: 3 = owner identification slow, change-log lookup;; 2 = slow;; 1 = vague;; 0 = none.
- EX026 — Test copy. Why pilot on a test copy? Rubric: 3 = no risk to production;; 2 = safety;; 1 = vague;; 0 = none.
Synthesis
The baseline replicates the size-error association and the pilot makes the procedure practical.
Continue studying — chapter 04 of 08
← Chapter 03 (draft) · Course chapter index · Chapter 05 (draft) →