Chapter 5 — Proposal and pilot
Lesson map
Lesson 05 published. Map, example, EX026.
flowchart TD
N0["Learning outcomes and connection"]
N1["5.1 Proposal structure"]
N2["5.2 Hypotheses"]
N3["5.3 Pilot criteria and run"]
N4["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4
N4 --> EX["Worked example"]
EX --> Q["Exercise EX026"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["5.1 Proposal structure"]
N2["5.2 Hypotheses"]
N3["5.3 Pilot criteria and run"]
N4["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX026"]
[Learning outcomes and connection]
|
v
[5.1 Proposal structure]
|
v
[5.2 Hypotheses]
|
v
[5.3 Pilot criteria and run]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX026]
Learning outcomes and connection
You will (1) write a dissertation proposal with aims, hypotheses and an analysis plan; (2) set pilot success criteria; (3) run a small testbed pilot safely; and (4) revise the protocol on the basis of the pilot. It turns the rivals of Chapter 4 into a plan and prepares data collection in Chapter 6.
5.1 Proposal structure
A master's proposal fits in eight to twelve pages: background and gap (Chapters 1–2); aims; hypotheses; design and data (Chapter 3); rivals and how each is addressed (Chapter 4); analysis plan; ethics and data management; timeline and risks. The Fragment Study's aims: (A1) estimate the change in failure rate attributable to offering X25519MLKEM768, by path; (A2) determine in a testbed whether the legacy proxy's failures depend on ClientHello size, on the codepoint, or on both; (A3) recommend a remediation that preserves quantum protection.
5.2 Hypotheses
- H1 — On the legacy-proxy path, failures per 10,000 series rise by at least 10 when the hybrid group is offered, relative to the concurrent control cohort; on the direct path the rise is below 2.
- H2 — In the testbed, proxy failures occur when and only when the ClientHello spans more than one TCP segment, independent of codepoint.
- H3 — After the proxy's firmware update or a bypass for TLS 1.3 inspection, the proxy-path excess disappears (difference below 2 per 10,000).
H2 is the decisive mechanism test; if failures also occur with a one-segment hello carrying the codepoint, H2 is rejected in favour of rival 3.
5.3 Pilot criteria and run
A two-week pilot on 2,000 synthetic series per cohort plus a testbed copy of the proxy (an isolated lab instance, never the production device) tests whether the pipeline works. Criteria: minimization verified on 100 records by a second person; series construction matches manual reconstruction on 50 cases; testbed can produce one-segment and two-segment hellos with the same codepoint by adjusting padding; coders reach κ ≥ 0.75 on 40 failure logs. The synthetic pilot meets all but the series criterion: 6 of 50 cases differed because retries to a different IP address of the same host were treated as new series. The rule is amended to define the series by hostname rather than address.
5.4 Worked example: revising after the pilot
The revision memo states the problem (series split by address), its effect (denominators inflated by about 4% on the proxy path), the amendment (hostname-based series with a 10-second window), and a re-check (50 new cases, 50/50 match). It also records a decision not to change: the coders asked to add a label for "certificate pinning", but there were only two such cases, so they remain under "cert" with a note. Pilots should produce small, documented changes, not a new study.
Attach the memo to the protocol as a dated amendment.
5.5 Common errors and limits
- Hypotheses without a threshold, so that any rise "supports" them.
- Running size or codepoint experiments on the production proxy.
- Treating a pilot as preliminary results to report as findings.
- Amending the protocol silently.
Each error has a guard in the Fragment proposal. H1 and H3 have numeric thresholds per 10,000 series. All size and codepoint experiments use the isolated testbed instance. Pilot data appear only in the methods chapter as checks on the pipeline. Every amendment, such as the hostname-based series rule, is written down with its date and reason.
Exercises — chapter 5
- EX026 — Aims to hypotheses. Write one hypothesis per aim with a threshold. Rubric: 3 = each falsifiable.
- EX027 — Decisive test. Explain why H2 is the decisive mechanism test and what result rejects it. Rubric: 3 = links to rival 3.
- EX028 — Pilot criteria. Add two pilot criteria of your own. Rubric: 3 = measurable, linked to a risk.
- EX029 — Revision memo. Write the memo for a pilot where κ was 0.66. Rubric: 3 = cause, amendment, re-test.
- EX030 — Risk register. List five project risks with likelihood, impact and mitigation. Rubric: 3 = includes data access and ethics delay.
- EX031 — Timeline. Draft a 24-month Gantt-style list for the dissertation. Rubric: 3 = realistic, includes validation on Fenwick data.
Chapter synthesis
A proposal turns rivals into falsifiable hypotheses with a decisive mechanism test; a pilot checks the pipeline against stated criteria and produces documented, minimal amendments.
Continue studying — chapter 05 of 08
← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →