Chapter 3 — Original contribution proposal
Lesson map
Lesson 03 published. Map, example, EX015.
flowchart TD
N0["Learning outcomes"]
N1["3.1 Hypotheses"]
N2["3.2 The review procedure"]
N3["3.3 Design and comparison"]
N4["3.4 Threats to validity"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX015"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["3.1 Hypotheses"]
N2["3.2 The review procedure"]
N3["3.3 Design and comparison"]
N4["3.4 Threats to validity"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX015"]
[Learning outcomes]
|
v
[3.1 Hypotheses]
|
v
[3.2 The review procedure]
|
v
[3.3 Design and comparison]
|
v
[3.4 Threats to validity]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX015]
Learning outcomes
You will (1) state hypotheses; (2) design the replication and the intervention study; (3) plan analysis and threats to validity.
3.1 Hypotheses
Ines states two hypotheses before seeing partner data. H1: across firewalls, rule-base size is positively correlated with the number of coded errors. H2: firewalls whose teams follow the structured review procedure for one quarter show fewer errors per hundred rules than they did before, and fewer than a comparison group reviewing in their usual way. Each hypothesis names its measure and test, and both are registered with her supervisor and the department's research office.
Each hypothesis is written before data collection and registered with her supervisor.
3.2 The review procedure
The structured review is a short, repeatable procedure. For each rule, the team confirms an owner and documented purpose, checks for shadowing with an analysis tool, compares the rule's scope with its purpose, and checks hit counts over ninety days. Unclear rules are flagged for owner confirmation rather than deleted. The procedure is written as a one-page checklist so that it can be followed by any competent engineer, which is essential if the result is to generalise beyond Ines's partners.
3.3 Design and comparison
Twelve firewalls across the three partners are coded at baseline for H1. For H2, eight firewalls whose teams volunteer adopt the procedure, and their before-and-after rates are compared, with the other four as a comparison group. Random assignment is not possible because partners choose which teams take part, so Ines records team size, experience and workload to discuss selection effects. She also measures how long each review takes, because a procedure too costly to repeat would not be adopted.
The design is documented before data is seen.
3.4 Threats to validity
Ines lists threats and responses. Coders might know which firewalls were reviewed, so a second coder, blind to group, recodes a sample. Teams might change behaviour simply because they are observed; the comparison group helps estimate this. Rule bases change for reasons unrelated to the study, such as new projects; change logs are used to separate these. Finally, three partners in one sector may not represent others, a limit stated plainly in the proposal.
Each threat is paired with the mitigation planned and any that remain.
Worked example
The proposal fits on eight pages: question, hypotheses, procedure, design, measures, analysis plan, threats and ethics. The committee asks for a blinded recoding sample, which Ines adds.
Synthetic case. The proposal and partners are synthetic.
Errors and limits
Without registered hypotheses, a study can drift toward whichever result looks strongest.
Exercises
- EX015 — Hypothesis wording. Rewrite H2 so it names measure, groups and period. Rubric: 3 = all three;; 2 = two;; 1 = vague;; 0 = none.
- EX016 — Owner confirmation. Why are unclear rules flagged rather than deleted? Rubric: 3 = avoid outages, confirm purpose;; 2 = safety;; 1 = vague;; 0 = none.
- EX017 — Selection effects. Why record team size and experience? Rubric: 3 = discuss non-random selection;; 2 = context;; 1 = vague;; 0 = none.
- EX018 — Blinding. What does a blinded recoding sample protect against? Rubric: 3 = coder expectation bias;; 2 = bias;; 1 = vague;; 0 = none.
- EX019 — Review time. Why measure how long reviews take? Rubric: 3 = adoption cost;; 2 = cost;; 1 = vague;; 0 = none.
- EX020 — Generalisation limit. State one limit on generalising from three partners. Rubric: 3 = same sector, similar practices;; 2 = limited;; 1 = vague;; 0 = none.
Synthesis
A registered proposal with a simple procedure, a comparison group and stated threats makes the contribution testable.
Continue studying — chapter 03 of 08
← Chapter 02 (draft) · Course chapter index · Chapter 04 (draft) →