Chapter 3 — Study design, ethics and reproducibility
Lesson map
Lesson 03 published. Map, example, EX015.
flowchart TD
N0["Learning outcomes"]
N1["3.1 Simulation design"]
N2["3.2 The reachability model"]
N3["Lab — Reachability under three policies"]
N4["3.3 Ethics with a real partner"]
N5["3.4 Reproducibility plan"]
N6["3.5 Pre-specifying analysis"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX015"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["3.1 Simulation design"]
N2["3.2 The reachability model"]
N3["Lab — Reachability under three policies"]
N4["3.3 Ethics with a real partner"]
N5["3.4 Reproducibility plan"]
N6["3.5 Pre-specifying analysis"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX015"]
[Learning outcomes]
|
v
[3.1 Simulation design]
|
v
[3.2 The reachability model]
|
v
[Lab — Reachability under three policies]
|
v
[3.3 Ethics with a real partner]
|
v
[3.4 Reproducibility plan]
|
v
[3.5 Pre-specifying analysis]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX015]
Learning outcomes
You will (1) design a simulation study with controlled comparisons; (2) build a reachability model on synthetic networks; (3) address ethics when a real organisation shapes the model; (4) plan reproducibility from the start.
3.1 Simulation design
Kenji designs his study as a controlled simulation. He generates many synthetic networks whose structure resembles Vantor's: departments of workstations, shared servers, a few administrative jump hosts and a domain controller. For each network he applies three policies: flat, zone-based with department zones and identity-based micro-segmentation. The outcome is the number of servers reachable from a randomly chosen compromised workstation. Generating many networks rather than one guards against conclusions that depend on a single arbitrary layout, and comparing all three policies on the same network removes variation that has nothing to do with the policy.
3.2 The reachability model
A network is modelled as a directed graph: an edge from host A to host B means policy allows A to connect to B. An attacker on a workstation can reach anything connected by a path, assuming each hop is exploitable. That assumption is deliberately pessimistic, giving an upper bound on reach. The lab below builds one small synthetic network under the three policies and counts reachable servers with a breadth-first search. The results show the pattern Kenji expects: flat access reaches every server, zones reach fewer, identity-based rules fewer still, but jump hosts reopen paths if their rules are broad.
Lab — Reachability under three policies
# Count servers reachable from one compromised workstation under three synthetic policies (BFS).
from collections import deque
ws = ["ws-fin1", "ws-fin2", "ws-ops1", "ws-ops2"]
srv = {"erp": "fin", "payroll": "fin", "wms": "ops", "fleet": "ops", "files": "all", "dc": "all"}
dept = lambda h: h[3:6]
def edges(policy, broad_jump=False):
E = {h: set() for h in ws + list(srv) + ["jump"]}
for w in ws:
for s, d in srv.items():
if policy == "flat" or (policy == "zone" and d in (dept(w), "all")) \
or (policy == "identity" and s == {"fin": "erp", "ops": "wms"}[dept(w)]):
E[w].add(s)
if policy == "identity": E[w].add("jump")
if broad_jump: E["jump"] |= set(srv)
return E
def reach(E, start):
seen, q = {start}, deque([start])
while q:
for n in E[q.popleft()]:
if n not in seen: seen.add(n); q.append(n)
return len([n for n in seen if n in srv])
for p in ["flat", "zone", "identity"]:
print(f"{p:9} servers reachable from ws-fin1: {reach(edges(p), 'ws-fin1')}")
print(f"identity + broad jump-host rule: {reach(edges('identity', True), 'ws-fin1')}")
Run with python3 lab_reach.py. Verified output:
flat servers reachable from ws-fin1: 6
zone servers reachable from ws-fin1: 4
identity servers reachable from ws-fin1: 1
identity + broad jump-host rule: 6
3.3 Ethics with a real partner
Although the networks are synthetic, they are shaped by Vantor's real structure, which is sensitive information. Kenji obtains his employer's written permission, agrees that no real hostnames, addresses or rules will appear in the dissertation and abstracts the structure into parameters such as department counts and server ratios. His ethics application states that no human participants are involved and that no testing touches Vantor systems. He also considers dual use: a detailed map of where Vantor is weakest would help attackers, so the dissertation reports only aggregate patterns across generated networks.
3.4 Reproducibility plan
From the first day, Kenji keeps code in version control, fixes random seeds for every experiment and records the parameters of each run in a configuration file stored alongside the results. He writes a short script that reruns all experiments from scratch and checks that summary numbers match the stored ones. He records software versions. Dr Aydın asks him to try running the whole pipeline on a clean machine halfway through the project, which reveals one missing dependency. Planning reproducibility early is far cheaper than reconstructing it before submission, when memory of past decisions has faded.
3.5 Pre-specifying analysis
Before running the main experiments, Kenji writes his analysis plan: the primary comparison is identity-based versus zone-based mean reach, with a bootstrap confidence interval; secondary measures are rule count and the effect of jump-host rule breadth. He commits the plan in his repository with a timestamp. Exploratory analyses are allowed later but must be labelled. This discipline matters even in simulation, because with many parameters it is easy to search until some setting shows a striking difference and then present it as if it had been the plan all along.
Worked example
Kenji designs a three-policy simulation on generated networks, runs the reachability lab, writes an ethics plan that abstracts Vantor's structure and commits an analysis plan with seeds and configurations.
Synthetic case. Vantor's structure parameters and the lab network are synthetic.
Errors and limits
Assuming every hop is exploitable overstates reach; real attackers face patching and detection. Generated networks may miss unusual real topologies.
Exercises
- EX015 — Same-network comparison. Why apply all three policies to the same generated network? Rubric: 3 = removes variation unrelated to policy;; 2 = control;; 1 = no reason;; 0 = none.
- EX016 — Edge meaning. What does an edge from A to B mean in the model? Rubric: 3 = policy allows A to connect to B;; 2 = allowed connection;; 1 = physical cable;; 0 = none.
- EX017 — Pessimistic bound. Why is assuming every hop exploitable called pessimistic? Rubric: 3 = it gives an upper bound on attacker reach;; 2 = upper bound;; 1 = optimistic;; 0 = none.
- EX018 — Lab reach counts. What reachable-server counts did the lab print for flat, zone and identity policies? Rubric: 3 = flat 6, zone 4, identity 1;; 2 = two correct;; 1 = wrong;; 0 = none.
- EX019 — Jump host effect. What did the lab show when the jump host's rules were broad? Rubric: 3 = reach rose again to 6, reopening paths;; 2 = rose;; 1 = no change;; 0 = none.
- EX020 — Dual-use concern. How does Kenji handle the dual-use risk of his findings? Rubric: 3 = reports only aggregate patterns, no Vantor-specific weak points;; 2 = aggregate;; 1 = publishes map;; 0 = none.
Synthesis
A sound simulation study controls comparisons, states pessimistic assumptions, protects partner information and fixes its analysis and seeds in advance.
Continue studying — chapter 03 of 08
← Chapter 02 (draft) · Course chapter index · Chapter 04 (draft) →