Chapter 3 — Study design, ethics and reproducibility
Lesson map
Lesson 03 published. Map, example, EX015.
flowchart TD
N0["Learning outcomes"]
N1["3.1 Design"]
N2["3.2 Ethics and data"]
N3["3.3 Analysis plan"]
N4["3.4 Reproducibility"]
N5["3.5 Change control"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX015"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["3.1 Design"]
N2["3.2 Ethics and data"]
N3["3.3 Analysis plan"]
N4["3.4 Reproducibility"]
N5["3.5 Change control"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX015"]
[Learning outcomes]
|
v
[3.1 Design]
|
v
[3.2 Ethics and data]
|
v
[3.3 Analysis plan]
|
v
[3.4 Reproducibility]
|
v
[3.5 Change control]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX015]
Learning outcomes
You will (1) design a before-and-after study with comparison; (2) handle staff data ethically; (3) make the analysis reproducible.
3.1 Design
Halyard enrols departments in passkeys over three months, so Jonah uses a staggered design. He compares each department's reset rate in six weeks before enrolment with six weeks after, and uses departments not yet enrolled as a comparison over the same calendar weeks. If reset rates fall only in enrolled departments, passkeys are a more plausible cause than a general trend. The order of departments is set by Halyard's IT team for practical reasons, not chosen by Jonah, and he records that order in advance, because it cannot be randomised.
3.2 Ethics and data
Sign-in and ticket data reveal individuals' working patterns. Jonah's data agreement gives him only weekly aggregates per department, never individual sign-ins or names, prepared by Halyard's identity team. Departments with fewer than ten staff are merged so no individual can be singled out. Data stays on a university server with access logged and is deleted after the degree is awarded. Ethics approval covers short interviews with desk staff, who can withdraw. Jonah never accesses Halyard's systems himself, and the examples in this book use synthetic data with the same structure.
3.3 Analysis plan
Before seeing data, Jonah writes his analysis plan. The main comparison is the change in reset rate per hundred staff in enrolled departments minus the change in not-yet-enrolled departments over the same weeks. Sign-in success is compared with an interval for each method. He specifies what would count against his hypothesis, a reset reduction similar in both groups or a fall in sign-in success, and how he will handle missing weeks. Writing this first prevents him from choosing, after the fact, whichever analysis gives the most attractive result.
3.4 Reproducibility
Jonah keeps every script, the analysis plan, data dictionary and version history in a repository. Scripts use only the Python standard library and read a single aggregate file whose format is documented, so anyone with a file of the same structure can rerun them. A synthetic file with that structure is published with the dissertation. He records the exact export date and query used by Halyard's team. When the dissertation reports a number, a reader can trace it to the script and input that produced it, which is what reproducible means in practice.
3.5 Change control
Because Halyard's rollout is a real change, Jonah has no say over its pace and must not influence it to suit his study. He agrees with the IT team that any change to the rollout plan, such as pausing a department after problems, will be shared with him and recorded. Such events become part of the data, not something to hide. Separating the organisation's operational decisions from the researcher's analysis protects both: Halyard runs its rollout safely, and Jonah's results honestly describe what actually happened.
Worked example
Jonah fixes a staggered before-and-after design with not-yet-enrolled departments as comparison, aggregate-only data, a written analysis plan and a reproducible script repository.
Synthetic case. Departments, data and agreements are synthetic.
Errors and limits
A simple before-and-after comparison without a comparison group cannot separate the effect of passkeys from seasonal or organisation-wide trends.
Exercises
- EX015 — Comparison group. What serves as the comparison group? Rubric: 3 = departments not yet enrolled, same calendar weeks;; 2 = not enrolled;; 1 = vague;; 0 = none.
- EX016 — Small-group rule. How are small departments handled? Rubric: 3 = merged if fewer than ten staff;; 2 = merged;; 1 = excluded;; 0 = none.
- EX017 — Main comparison. What is the main comparison in the analysis plan? Rubric: 3 = enrolled change minus not-yet-enrolled change in reset rate;; 2 = difference of changes;; 1 = before/after;; 0 = none.
- EX018 — Falsifying result. What result would count against the hypothesis? Rubric: 3 = similar reduction in both groups or lower sign-in success;; 2 = one;; 1 = vague;; 0 = none.
- EX019 — Traceable numbers. What makes a reported number reproducible here? Rubric: 3 = traceable to script and input, synthetic file published;; 2 = traceable;; 1 = vague;; 0 = none.
- EX020 — Rollout changes. How are changes to Halyard's rollout treated? Rubric: 3 = shared, recorded and included as data;; 2 = recorded;; 1 = hidden;; 0 = none.
Synthesis
A staggered design, aggregate data, a written plan and reproducible scripts make a workplace study both ethical and credible.
Continue studying — chapter 03 of 08
← Chapter 02 (draft) · Course chapter index · Chapter 04 (draft) →