Chapter 3 — Original contribution proposal
Lesson map
Lesson 03 published. Map, example, EX015.
flowchart TD
N0["Learning outcomes"]
N1["3.1 Hypotheses"]
N2["3.2 Measures"]
N3["3.3 Design"]
N4["3.4 Analysis plan"]
N5["3.5 Registration"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX015"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["3.1 Hypotheses"]
N2["3.2 Measures"]
N3["3.3 Design"]
N4["3.4 Analysis plan"]
N5["3.5 Registration"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX015"]
[Learning outcomes]
|
v
[3.1 Hypotheses]
|
v
[3.2 Measures]
|
v
[3.3 Design]
|
v
[3.4 Analysis plan]
|
v
[3.5 Registration]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX015]
Learning outcomes
You will (1) specify hypotheses and measures; (2) design a randomised comparison; (3) register the plan before data collection.
3.1 Hypotheses
Lucía states her primary hypothesis: contracts reviewed with a structured checklist derived from EthTrust requirements and the SWC categories will have fewer residual high-severity issues than contracts reviewed in the usual unstructured way. Residual issues are those found by a blinded independent reviewer after the first review has closed. A secondary hypothesis is that the checklist will reduce residual access-control issues most, because the baseline will show they dominate. Writing hypotheses before data collection prevents choosing them afterwards to fit the results.
3.2 Measures
The primary measure is the count of residual high-severity issues per contract. Severity follows a written rubric based on impact on funds, control and availability, with two raters and agreement checked. Secondary measures include review hours, findings per hour and the share of findings in each category. Lucía also records contract size and complexity as covariates. Each measure has a definition, a source and a planned analysis, so nothing is invented after the data arrive. Raters are trained on the rubric with ten practice findings first.
3.3 Design
Twenty-four synthetic but realistic contracts, prepared with partner engineers and seeded only with weaknesses documented in public registries, are randomly assigned to the two procedures, twelve each, within size strata. Reviewers are partner auditors trained in both procedures and assigned to avoid reviewing the same contract twice. The blinded independent reviewer does not know which procedure was used. Random assignment and blinding protect internal validity against contract difficulty and reviewer expectations. Lucía documents the allocation with a fixed seed so it can be checked later by examiners.
3.4 Analysis plan
Because counts are small and not normally distributed, Lucía plans a one-sided permutation test on the difference in mean residual high-severity issues, with a fixed random seed and ten thousand shuffles, plus the count of contracts with no residual high issues. She also plans a sensitivity analysis excluding the largest contracts. Effect sizes are reported with intervals, not only p-values. A short plan written now prevents running many analyses and reporting only the favourable one. The plan also states how missing data will be handled.
3.5 Registration
Lucía registers the hypotheses, measures, design and analysis plan with a timestamped document held by her supervisor and the ethics board before any review begins. Any later change will be reported as a deviation with its reason. Registration is the main safeguard against researcher degrees of freedom: it makes her commitments visible and lets examiners compare what was planned with what was done, which is particularly important when a candidate hopes for a particular result. The registered document is cited in every later chapter.
Worked example
The registered plan states two hypotheses, a residual-issue measure with a severity rubric, 24 contracts randomised within strata, blinded checking and a permutation analysis with fixed seed.
Synthetic case. Contracts, reviewers and plans are synthetic; EEA EthTrust and the SWC registry are real sources.
Errors and limits
Choosing hypotheses or analyses after seeing the data produces results that cannot be trusted.
Exercises
- EX015 — Residual definition. What is a residual issue? Rubric: 3 = high-severity issue found by blinded reviewer after first review closed;; 2 = missed issue;; 1 = any finding;; 0 = none.
- EX016 — Secondary hypothesis. What is the secondary hypothesis? Rubric: 3 = checklist reduces residual access-control issues most;; 2 = access control;; 1 = vague;; 0 = none.
- EX017 — Contract allocation. How are contracts allocated? Rubric: 3 = 24 randomised within size strata, 12 per arm;; 2 = random;; 1 = chosen;; 0 = none.
- EX018 — Seeding limit. What weaknesses may contracts contain? Rubric: 3 = only those documented in public registries;; 2 = public;; 1 = new ones;; 0 = none.
- EX019 — Planned test. What test is planned? Rubric: 3 = one-sided permutation test with fixed seed;; 2 = permutation;; 1 = t-test only;; 0 = none.
- EX020 — Registration purpose. What does registration guard against? Rubric: 3 = researcher degrees of freedom;; 2 = flexibility;; 1 = cost;; 0 = none.
Synthesis
Clear hypotheses, defined measures, randomisation, blinding and registration make the contribution testable.
Continue studying — chapter 03 of 08
← Chapter 02 (draft) · Course chapter index · Chapter 04 (draft) →