Chapter 3 — Study design, ethics and reproducibility
Lesson map
Lesson 03 published. Map, example, EX015.
flowchart TD
N0["Learning outcomes"]
N1["3.1 Choosing a design"]
N2["3.2 Ethics of studying requests"]
N3["3.3 Reproducible analysis"]
N4["3.4 Defining complexity"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX015"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["3.1 Choosing a design"]
N2["3.2 Ethics of studying requests"]
N3["3.3 Reproducible analysis"]
N4["3.4 Defining complexity"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX015"]
[Learning outcomes]
|
v
[3.1 Choosing a design]
|
v
[3.2 Ethics of studying requests]
|
v
[3.3 Reproducible analysis]
|
v
[3.4 Defining complexity]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX015]
Learning outcomes
You will (1) choose a quasi-experimental design for an operational change; (2) plan ethics for research on rights requests; (3) set up a reproducible analysis.
3.1 Choosing a design
Brookfield will launch the portal for all customers on one date, so Hana cannot randomise customers. She chooses an interrupted comparison: 240 requests in the six months before launch and 240 in the six months after. Shadish, Cook and Campbell (2002) warn that such designs are vulnerable to history, maturation and changes in who is studied, so she plans to measure the request mix in both periods. She also records any other change at Brookfield during the study, such as staffing, so that it can be considered as an alternative explanation rather than discovered too late.
3.2 Ethics of studying requests
Rights requests contain sensitive personal data and sometimes reveal disputes. Hana's ethics application to Ellery and Brookfield's data protection officer limits her access to ticket metadata: channel, dates, complexity and outcome. She never reads request content. Brookfield extracts metadata under its own legitimate interest in improving compliance, documented in its records, and Hana receives a pseudonymised extract. She states that she will not attempt to link tickets to customers. Every result is reported in aggregate only, and categories with fewer than ten requests are merged so that no individual can be inferred.
3.3 Reproducible analysis
Hana writes her analysis in a script with a fixed random seed and keeps it in version control. Every figure in the dissertation is generated by the script rather than typed by hand. She stores the pseudonymised extract inside Brookfield's environment and runs the script there, exporting only aggregate tables. This arrangement means that an examiner can inspect the code and rerun it on synthetic data with the same structure, while the real extract never leaves the company. Dr. Pritchard calls this "bringing the code to the data" and asks her to describe it in the methods chapter.
3.4 Defining complexity
Because complex requests take longer, Hana must define complexity before seeing results. She classes a request as complex if it involves more than one system holding the person's data, a third-party processor, or a request for a copy of call recordings. Two staff members classify a sample independently to check agreement. The definition is recorded in the protocol. She explains that if complexity were defined after the results, it could be adjusted to favour the portal, which would undermine the whole design and leave her conclusions open to a charge of convenient post-hoc categories.
Worked example
Hana chooses an interrupted before-and-after design with 240 requests per period, limits herself to pseudonymised metadata analysed inside Brookfield and defines complexity before seeing results.
Synthetic case. The design and extract are synthetic; Shadish, Cook and Campbell (2002) is real.
Errors and limits
Interrupted designs cannot rule out every alternative; measuring the request mix only reduces one threat.
Exercises
- EX015 — Why not randomise. Why can Hana not randomise customers to the portal? Rubric: 3 = Brookfield launches it for everyone on one date;; 2 = mentions launch;; 1 = says randomisation is unethical;; 0 = none.
- EX016 — Validity threats. Name two threats Shadish, Cook and Campbell associate with such designs. Rubric: 3 = history and changes in who is studied (also maturation);; 2 = names one;; 1 = names none;; 0 = none.
- EX017 — Metadata only. What data may Hana see? Rubric: 3 = pseudonymised ticket metadata: channel, dates, complexity, outcome;; 2 = lists some;; 1 = says full tickets;; 0 = none.
- EX018 — Small categories. Why merge categories with fewer than ten requests? Rubric: 3 = to avoid inferring individuals from small counts;; 2 = mentions privacy;; 1 = omits;; 0 = none.
- EX019 — Code to data. What does "bringing the code to the data" mean here? Rubric: 3 = the script runs inside Brookfield and only aggregates leave;; 2 = mentions location;; 1 = exports data;; 0 = none.
- EX020 — Pre-defined complexity. Why define complexity before seeing results? Rubric: 3 = to stop the definition being tuned to favour the portal;; 2 = mentions bias;; 1 = says it is tidy;; 0 = none.
Synthesis
Chapter 3 fixes design, ethics and reproducibility. Chapter 4 lists competing explanations for any improvement.
Continue studying — chapter 03 of 08
← Chapter 02 (draft) · Course chapter index · Chapter 04 (draft) →