Chapter 6 — Data collection and analysis
Lesson map
Lesson 06 published. Map, example, EX033.
flowchart TD
N0["Learning outcomes"]
N1["6.1 Running the main study"]
N2["6.2 Bootstrap comparison"]
N3["Lab — Bootstrap comparison of policies"]
N4["6.3 Rival checks"]
N5["6.4 The complexity trade-off"]
N6["6.5 Presenting results honestly"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX033"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["6.1 Running the main study"]
N2["6.2 Bootstrap comparison"]
N3["Lab — Bootstrap comparison of policies"]
N4["6.3 Rival checks"]
N5["6.4 The complexity trade-off"]
N6["6.5 Presenting results honestly"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX033"]
[Learning outcomes]
|
v
[6.1 Running the main study]
|
v
[6.2 Bootstrap comparison]
|
v
[Lab — Bootstrap comparison of policies]
|
v
[6.3 Rival checks]
|
v
[6.4 The complexity trade-off]
|
v
[6.5 Presenting results honestly]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX033]
Learning outcomes
You will (1) run the main simulation and collect results reproducibly; (2) compare policies with bootstrap confidence intervals; (3) evaluate rival checks; (4) present results without overstating them.
6.1 Running the main study
Kenji runs two hundred networks at each of three sizes, applying the flat, zone, identity and random-edge policies, with the rival parameters varied as planned. Every run writes its configuration, seed and results to a file, and the rerun script confirms that a random sample of runs reproduces exactly. He resists the urge to look at results while runs are still incomplete, because early peeking tends to shape later decisions. When all runs finish, he computes the pre-specified primary comparison first, before any exploratory analysis, and records the time he did so.
6.2 Bootstrap comparison
The primary comparison is the mean difference in reachable servers between zone and identity policies on the same networks. Kenji uses a bootstrap: he resamples networks with replacement many times, recomputes the mean difference each time and takes the middle ninety-five per cent of those values as a confidence interval. The bootstrap needs few assumptions about the shape of the data. The lab below runs this procedure on synthetic paired results for one network size and also compares identity with the random-edge control, reporting both intervals and the mean rule counts.
Lab — Bootstrap comparison of policies
# Bootstrap CIs for paired synthetic results (servers reachable) across 200 generated networks.
import random, statistics
random.seed(42)
nets = []
for _ in range(200):
zone = random.randint(4, 9); ident = max(0, zone - random.choice([1, 2, 2, 3, 3, 4]))
ctrl = max(ident, ident + random.choice([0, 1, 2, 2, 3]))
nets.append((zone, ident, ctrl, random.randint(18, 26), random.randint(60, 75)))
def boot(diffs, B=4000):
m = [statistics.mean(random.choices(diffs, k=len(diffs))) for _ in range(B)]; m.sort()
return statistics.mean(diffs), m[int(.025 * B)], m[int(.975 * B)]
for name, i in [("zone - identity", 0), ("control - identity", 2)]:
est, lo, hi = boot([n[i] - n[1] for n in nets])
print(f"{name:20} mean {est:.2f} 95% CI [{lo:.2f}, {hi:.2f}]")
print(f"mean rules: zone {statistics.mean(n[3] for n in nets):.1f}, identity {statistics.mean(n[4] for n in nets):.1f}")
Run with python3 lab_compare.py. Verified output:
zone - identity mean 2.43 95% CI [2.29, 2.56]
control - identity mean 1.56 95% CI [1.42, 1.71]
mean rules: zone 22.2, identity 67.7
6.3 Rival checks
The random-edge control reaches noticeably more servers than identity, so structure, not just edge count, explains much of the benefit. The advantage holds across network sizes, though it shrinks for the smallest remaining size. Allowing identities to reach up to four servers reduces but does not remove the advantage. Template copying is the strongest rival: when a third of workstations inherit a broad rule set, most of the advantage disappears. Misconfiguration at five per cent narrows the gap but leaves it above Kenji's overturn threshold. He reports each rival check in a table with its result.
6.4 The complexity trade-off
Identity policies need many more allow rules than zone policies: in the lab, roughly three times as many on average. Kenji presents reach and rule count side by side, so readers can see the trade-off rather than only the security gain. He does not claim to know how many rules an organisation can manage, but he notes that the template-copy result suggests organisations often cope with rule complexity by copying, which undermines the benefit. This connects the quantitative results back to the economy-of-mechanism principle discussed in the seminar.
6.5 Presenting results honestly
Kenji's results chapter states effects with intervals, explains what each check showed and labels any exploratory analysis. He avoids saying identity segmentation is secure or that zone segmentation is insecure. He writes that, under the model's assumptions, identity-based segmentation reduced worst-case reach from a single compromised workstation compared with zone segmentation, that much of this gain depended on avoiding template-copied broad rules, and that it required about three times as many rules. That sentence is longer than a headline, but every word is supported by a reported result.
Worked example
Kenji runs six hundred networks, computes bootstrap intervals for zone minus identity and random control minus identity, reports rival checks in a table and presents reach with rule counts.
Synthetic case. All results are from synthetic simulations.
Errors and limits
Bootstrap intervals describe variation across generated networks, not uncertainty about real enterprises. Exploratory findings need confirmation.
Exercises
- EX033 — Early peeking. Why avoid looking at results before runs finish? Rubric: 3 = early peeking tends to shape later decisions;; 2 = bias;; 1 = no reason;; 0 = none.
- EX034 — Bootstrap steps. Describe the bootstrap Kenji uses. Rubric: 3 = resample networks with replacement, recompute mean difference, take middle 95%;; 2 = partial;; 1 = vague;; 0 = none.
- EX035 — Lab intervals. What mean differences did the lab report for zone minus identity and control minus identity? Rubric: 3 = about 2.4 and 1.6 servers, both intervals above zero;; 2 = one;; 1 = wrong;; 0 = none.
- EX036 — Strongest rival. Which rival check removed most of the advantage? Rubric: 3 = template copying of broad rules for a third of workstations;; 2 = template;; 1 = size;; 0 = none.
- EX037 — Rule ratio. Roughly how many more rules did identity policies need than zone policies in the lab? Rubric: 3 = about three times as many;; 2 = ratio;; 1 = fewer;; 0 = none.
- EX038 — Supported sentence. Why does Kenji avoid calling identity segmentation secure? Rubric: 3 = results only support reduced modelled reach under assumptions, with conditions;; 2 = conditional;; 1 = it is secure;; 0 = none.
Synthesis
Careful analysis reports pre-specified comparisons with intervals, tests every rival and keeps security gains and operational costs side by side.
Continue studying — chapter 06 of 08
← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →