Chapter 5 — Proposal and pilot
Lesson map
Lesson 05 published. Map, example, EX027.
flowchart TD
N0["Learning outcomes"]
N1["5.1 The proposal"]
N2["5.2 Pilot goals"]
N3["5.3 How many networks"]
N4["5.4 Adjusting the plan"]
N5["5.5 Pilot reporting"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX027"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["5.1 The proposal"]
N2["5.2 Pilot goals"]
N3["5.3 How many networks"]
N4["5.4 Adjusting the plan"]
N5["5.5 Pilot reporting"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX027"]
[Learning outcomes]
|
v
[5.1 The proposal]
|
v
[5.2 Pilot goals]
|
v
[5.3 How many networks]
|
v
[5.4 Adjusting the plan]
|
v
[5.5 Pilot reporting]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX027]
Learning outcomes
You will (1) write a concise dissertation proposal; (2) run a pilot to test code and parameters; (3) estimate how many simulated networks are needed; (4) adjust the plan from pilot evidence.
5.1 The proposal
Kenji's proposal fits on six pages. It states the question, the three policies and the random-edge control, the outcomes of mean reach and rule count, the rival checks and overturn thresholds, the ethics agreement with Vantor and the timeline. It lists deliverables: a dissertation, a code repository with fixed seeds and a short practitioner brief. Dr Aydın asks him to cut the background section in half and expand the methods, because reviewers judge feasibility by methods. The committee approves it with one condition: a pilot must confirm that the simulation runs within the available computing time.
5.2 Pilot goals
The pilot runs the full pipeline on twenty small networks. Its goals are practical: confirm that the code produces sensible reach values, check the run time per network, find bugs and measure how much results vary between networks. It is not meant to answer the question. Kenji writes a test that compares the breadth-first search with a simple brute-force count on tiny networks where the right answer can be checked by hand, a known-answer test similar in spirit to test vectors. The test catches a bug in which jump-host edges were added twice, inflating reach in one policy.
5.3 How many networks
Using the pilot's variation, Kenji estimates how many networks he needs for the main comparison. The standard deviation of the difference between identity and zone reach across pilot networks is about two servers. To estimate the mean difference within about half a server with ninety-five per cent confidence, he needs roughly sixty networks per size. He chooses two hundred per size to allow for the extra parameters from the rival checks. Run time from the pilot shows this fits easily on a laptop overnight, so the committee's computing condition is met.
5.4 Adjusting the plan
The pilot also shows that at the smallest network size, zone and identity policies often give identical reach because each department has only one server. Kenji drops that size and adds a larger one, documenting the change and its reason in his plan's change log. He also discovers that the misconfiguration parameter at ten per cent makes all policies nearly flat, so he limits the range to zero to five per cent, a range the Vantor engineers consider realistic. Each adjustment is recorded before the main runs begin.
5.5 Pilot reporting
Kenji's pilot report is two pages: what was run, what was found, what changed and why. It includes the bug found by the known-answer test and the dropped network size. Dr Aydın stresses that the report should be honest about problems, because examiners trust a study more when they can see how it was debugged. The report becomes an appendix to the dissertation. Kenji also tags the code version used for the pilot in his repository, so the main experiments start from a known, tested state rather than from code still changing.
Worked example
Kenji writes a six-page proposal, pilots twenty networks, fixes a double-counted jump-host bug with a known-answer test, sizes the main study at two hundred networks per size and logs two plan adjustments.
Synthetic case. Pilot numbers and committee decisions are synthetic.
Errors and limits
Sample-size estimates from a small pilot are themselves uncertain. Dropping a network size changes the population the results describe.
Exercises
- EX027 — Committee condition. What condition did the committee attach to approval? Rubric: 3 = a pilot confirming the simulation fits the available computing time;; 2 = pilot;; 1 = vague;; 0 = none.
- EX028 — Known-answer bug. What bug did the known-answer test catch? Rubric: 3 = jump-host edges added twice, inflating reach;; 2 = jump-host;; 1 = vague;; 0 = none.
- EX029 — Networks needed. How many networks per size did the estimate require, and how many were chosen? Rubric: 3 = about sixty needed; two hundred chosen;; 2 = one figure;; 1 = wrong;; 0 = none.
- EX030 — Dropped size. Why was the smallest network size dropped? Rubric: 3 = one server per department made zone and identity reach identical;; 2 = reason;; 1 = vague;; 0 = none.
- EX031 — Misconfiguration range. What range was set for misconfiguration, and why? Rubric: 3 = zero to five per cent, considered realistic by engineers;; 2 = range;; 1 = ten per cent;; 0 = none.
- EX032 — Tagging pilot code. Why tag the pilot code version? Rubric: 3 = so main experiments start from a known tested state;; 2 = known state;; 1 = no reason;; 0 = none.
Synthesis
A short proposal and an honest pilot turn a research plan into a tested pipeline with a justified sample size.
Continue studying — chapter 05 of 08
← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →