Chapter 3 — Original contribution proposal
Lesson map
Lesson 03 published. Map, example, EX015.
flowchart TD
N0["Learning outcomes"]
N1["3.1 Hypotheses"]
N2["3.2 Measures"]
N3["3.3 Threats to validity"]
N4["3.4 Approval"]
N0 --> N1 --> N2 --> N3 --> N4
N4 --> EX["Worked example"]
EX --> Q["Exercise EX015"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["3.1 Hypotheses"]
N2["3.2 Measures"]
N3["3.3 Threats to validity"]
N4["3.4 Approval"]
N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX015"]
[Learning outcomes]
|
v
[3.1 Hypotheses]
|
v
[3.2 Measures]
|
v
[3.3 Threats to validity]
|
v
[3.4 Approval]
|
v
[Worked example] --> [Exercise EX015]
Learning outcomes
You will (1) state hypotheses and outcome measures precisely; (2) design comparison and threats to validity; (3) obtain approval for a proposal.
3.1 Hypotheses
Yusuf states two hypotheses. First, departments adopting risk-ranked patching will show a larger fall in median days to patch critical vulnerabilities than comparison departments over the same period. Second, departments adopting risk-ranked access reviews will show a larger fall in the share of accounts with privileges beyond their role baseline. He records the hypotheses, measures, analysis method and the minimum effect he would consider meaningful in a dated pre-registration document held by Prof. Hald. Prof. Hald signs and dates the document, and a copy is lodged with the graduate school.
3.2 Measures
Patch latency is the number of days from a critical vulnerability being published by the vendor to the fix being installed on an affected system, using medians because latency is skewed by a few very slow cases. Excess privilege is the share of accounts holding any permission outside their role's baseline, a direct operational measure of least privilege. Both come from existing systems, avoiding self-report bias. Yusuf documents each measure's source, formula and known errors, such as patches applied but not recorded.
3.3 Threats to validity
Yusuf lists threats. Departments chosen for early adoption might already be better run, a selection threat; he will compare pre-adoption trends. A citywide event, such as a major vendor vulnerability, might affect all departments; the comparison group absorbs it. Recording practices might change with adoption, a measurement threat; he will audit a sample of records against system evidence. Staff might change behaviour because they know they are studied; he notes this but expects little effect on automated logs. Each threat and response appears in the proposal.
3.4 Approval
The proposal panel includes Prof. Hald, a statistician and Fernvale's security lead as an external adviser. The statistician asks about sample size; Yusuf admits he cannot control how many vulnerabilities occur, and plans to report confidence intervals and power honestly rather than claiming certainty. The security lead asks that department names be coded in all outputs. The panel approves the proposal with both points recorded, and the pre-registration is frozen before outcome data are extracted. The security lead also offers to arrange access to anonymised test extracts.
Worked example
Yusuf pre-registers two hypotheses on patch latency and excess privilege, defines measures from system data, lists selection, history, measurement and observation threats and gains approval with commitments on power reporting and coded department names.
Synthetic case. The proposal is synthetic.
Errors and limits
Changing hypotheses after seeing outcome data invalidates confirmatory claims.
Exercises
- EX015 — First hypothesis. What does the first hypothesis predict? Rubric: 3 = a larger fall in median critical patch latency in adopting departments;; 2 = partly correct;; 1 = partial;; 0 = none.
- EX016 — Median choice. Why use medians for patch latency? Rubric: 3 = latency is skewed by a few very slow cases;; 2 = partly correct;; 1 = vague;; 0 = none.
- EX017 — Excess privilege definition. How is excess privilege measured? Rubric: 3 = share of accounts with permissions outside their role baseline;; 2 = partly correct;; 1 = vague;; 0 = none.
- EX018 — Selection response. How will Yusuf address the selection threat? Rubric: 3 = compare pre-adoption trends;; 2 = partly correct;; 1 = vague;; 0 = none.
- EX019 — Statistician question. What did the statistician ask about? Rubric: 3 = sample size;; 2 = partly correct;; 1 = other;; 0 = none.
- EX020 — Freeze timing. When is the pre-registration frozen? Rubric: 3 = before outcome data are extracted;; 2 = partly correct;; 1 = after;; 0 = none.
Continue studying — lesson 03 (chapter 03 of 08)
← Chapter 02 · Course chapter index · Chapter 04 →
Continue to the next authored lesson.