Chapter 3 — Study design, ethics and reproducibility
Lesson map
Lesson 03 published. Map, example, EX015.
flowchart TD
N0["Learning outcomes"]
N1["3.1 Choosing a design"]
N2["3.2 Checking assumptions"]
N3["3.3 Ethics"]
N4["3.4 Reproducible analysis"]
N0 --> N1 --> N2 --> N3 --> N4
N4 --> EX["Worked example"]
EX --> Q["Exercise EX015"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["3.1 Choosing a design"]
N2["3.2 Checking assumptions"]
N3["3.3 Ethics"]
N4["3.4 Reproducible analysis"]
N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX015"]
[Learning outcomes]
|
v
[3.1 Choosing a design]
|
v
[3.2 Checking assumptions]
|
v
[3.3 Ethics]
|
v
[3.4 Reproducible analysis]
|
v
[Worked example] --> [Exercise EX015]
Learning outcomes
You will (1) design a quasi-experimental evaluation; (2) address ethics in workplace research; (3) plan reproducible analysis.
3.1 Choosing a design
The council will not randomise which departments get stewards; it has chosen council tax and waste collection. Tomás therefore uses a difference-in-differences design. He compares the change in correction times in the two stewarded departments with the change in the three comparison departments over the same period. The design removes stable differences between departments and common trends, such as a council-wide software update, provided the departments would have followed parallel trends without the pilot. It cannot remove changes that affect only one group, such as a new manager in council tax.
3.2 Checking assumptions
Tomás examines twelve months of correction times before the pilot using a synthetic replica built from summary figures the council publishes. Trends in both groups look similar, which supports the parallel trends assumption, although it cannot prove it. He also lists threats: stewarded departments might receive extra attention, staff turnover might differ and a change in reporting might alter which errors are recorded. He plans to check each threat with council records and interviews rather than assuming it away. Each threat gets a check.
3.3 Ethics
The study involves council employees, whose performance might seem to be judged. Tomás's ethics application states that he analyses department-level times only, never individual staff performance, and that interview participants are anonymous and can withdraw. Residents' addresses are not used; the replica contains synthetic addresses with the same error patterns. The council's data protection officer confirms that no personal data leave the council. Dr. Solberg notes that workplace research needs particular care because power differences make refusal difficult. Participation is voluntary.
3.4 Reproducible analysis
Tomás writes the analysis plan before collecting data: the estimator, the confidence interval method, the bootstrap resampling count and the threshold for practical importance, which he sets at five days. All code uses the Python standard library, fixed random seeds and version control. He records the plan with a date so he cannot later change the outcome or threshold after seeing results. Reproducibility, he notes, is especially important for governance studies that councils might use to justify spending on new roles.
Worked example
Tomás chooses difference-in-differences, checks pre-pilot trends on a synthetic replica, lists threats, limits analysis to department level for ethics and fixes an analysis plan with a five-day importance threshold.
Synthetic case. The design is synthetic.
Errors and limits
Parallel trends can hold before and fail after; watch for new events.
Exercises
- EX015 — Why not randomise. Why can Tomás not randomise? Rubric: 3 = the council chose the pilot departments;; 2 = partly correct;; 1 = vague;; 0 = none.
- EX016 — Design name. Which design does Tomás use? Rubric: 3 = difference-in-differences;; 2 = partly correct;; 1 = other;; 0 = none.
- EX017 — Key assumption. What assumption does the design need? Rubric: 3 = parallel trends;; 2 = partly correct;; 1 = vague;; 0 = none.
- EX018 — Threat example. Name one threat to the design. Rubric: 3 = extra attention, turnover differences, reporting changes;; 2 = one;; 1 = vague;; 0 = none.
- EX019 — Ethics level. At what level is performance analysed? Rubric: 3 = department level only;; 2 = partly correct;; 1 = individual;; 0 = none.
- EX020 — Importance threshold. What threshold of practical importance was set? Rubric: 3 = five days;; 2 = partly correct;; 1 = wrong;; 0 = none.
Continue studying — lesson 03 (chapter 03 of 08)
← Chapter 02 · Course chapter index · Chapter 04 →
Continue to the next authored lesson.