◇ OpenCyber Academy
Doctoral research · Chapter 3 · English

Chapter 3 — Original contribution proposal

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 3 — Original contribution proposal

Lesson map

Lesson 03 published. Map, example, EX015.

flowchart TD
  N0["Learning outcomes"]
  N1["3.1 From question to hypotheses"]
  N2["3.2 Choosing the outcome"]
  N3["3.3 Design and power"]
  N4["3.4 Registration and analysis plan"]
  N5["3.5 Risks to the partner"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX015"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["3.1 From question to hypotheses"]
  N2["3.2 Choosing the outcome"]
  N3["3.3 Design and power"]
  N4["3.4 Registration and analysis plan"]
  N5["3.5 Risks to the partner"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX015"]

[Learning outcomes]
    |
    v
[3.1 From question to hypotheses]
    |
    v
[3.2 Choosing the outcome]
    |
    v
[3.3 Design and power]
    |
    v
[3.4 Registration and analysis plan]
    |
    v
[3.5 Risks to the partner]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX015]

Learning outcomes

You will (1) write falsifiable hypotheses for a misuse-reduction study; (2) choose a primary outcome and analysis before seeing data; (3) register the plan so later changes are visible.

3.1 From question to hypotheses

Nadia turns her question into a primary hypothesis and a secondary one. H1 states that teams using the checklist and lint step will average fewer confirmed misuse findings per ten merged changes than teams using usual review over one quarter. H2 states that a larger share of checklist teams will record zero misuse in that quarter. Both are directional and falsifiable: a null or reversed difference counts against them. She also writes a null expectation for the lint tool alone, because examiners will ask whether a cheap tool, rather than human attention, explains any effect.

3.2 Choosing the outcome

The primary outcome is confirmed misuse per ten merged changes that touch cryptographic code, measured per team. Confirmation means a blinded auditor, unaware of the arm, agrees that the change violates a codebook rule. Nadia rejects two tempting alternatives. Lint warnings would reward the tool for flagging its own patterns, and incidents would be too rare to measure in a quarter. She defines "touches cryptographic code" mechanically, as any change importing Meridian's approved crypto wrapper or the standard library modules hashlib, hmac or secrets, so that inclusion is not a judgement call.

3.3 Design and power

Twenty-four teams are available, so Nadia plans a cluster-randomised design with twelve teams per arm, stratified by product line so that tokenisation, webhooks and storage teams appear in both arms. Team-level randomisation avoids contamination, since engineers in one team share review habits. She estimates power from the baseline she will measure in Chapter 4 and states the honest limitation: with twelve clusters per arm only large effects are reliably detectable. She therefore pre-commits to a permutation test, which makes no normality assumption and suits small cluster counts.

3.4 Registration and analysis plan

Before any outcome data exist, Nadia deposits a time-stamped plan with her university's research office. It lists the hypotheses, primary and secondary outcomes, inclusion rule, randomisation seed policy, the permutation test with 20,000 shuffles, and the rule for handling teams that leave Meridian. It also lists what would count as a deviation. Registration is not bureaucracy for its own sake: it lets examiners distinguish confirmatory results from exploratory ones, and it protects Nadia from the temptation to switch outcomes when the first one disappoints.

3.5 Risks to the partner

The proposal also protects Meridian. The checklist never asks reviewers to test exploits; it asks questions such as whether each GCM key has an invocation counter and rotation trigger, whether keys come from a managed key service rather than source code, and whether password storage uses a memory-hard function such as Argon2 from RFC 9106. Control teams keep their usual review, so no team receives weaker protection than before. The escalation duty from Chapter 1 still applies in both arms, and the security lead may halt the study if any arm causes harm.

Worked example

Nadia's one-page registration states H1 and H2, the outcome per ten crypto-touching changes, the 12-versus-12 stratified cluster design and the permutation test, then deposits it with a timestamp before baseline data are unblinded.

Synthetic note. The registration office and checklist are invented for teaching; RFC 9106 and SP 800-38D are real references for the checklist items.

Errors and limits

Registration cannot fix a weak outcome measure, and a small cluster count limits power whatever the test. Report deviations openly rather than silently amending the plan.

Exercises

Synthesis

A registered plan with falsifiable hypotheses, a blinded and mechanically defined outcome and a design matched to the available clusters turns a good question into a test.

Continue studying — chapter 03 of 08

← Chapter 02 (draft) · Course chapter index · Chapter 04 (draft) →