Chapter 33 — Research methods II
Lesson map
Lesson 33 published. Map, example, EX097.
flowchart TD
N0["Learning outcomes"]
N1["33.1 Analysing the data"]
N2["Lab — Difference-in-differences on report rates"]
N3["33.2 Threats to validity"]
N4["33.3 Reporting with caution"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX097"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["33.1 Analysing the data"]
N2["Lab — Difference-in-differences on report rates"]
N3["33.2 Threats to validity"]
N4["33.3 Reporting with caution"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX097"]
[Learning outcomes]
|
v
[33.1 Analysing the data]
|
v
[Lab — Difference-in-differences on report rates]
|
v
[33.2 Threats to validity]
|
v
[33.3 Reporting with caution]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX097]
Learning outcomes
You will (1) analyse the quasi-experiment data from chapter 18; (2) address threats to validity; (3) report findings with appropriate caution.
33.1 Analysing the data
Lucas collects six months of aggregated report-button counts. Briefed stores show a monthly report rate per staff member before and after the briefings; comparison stores show the same over the same months. He computes the change in each group and the difference between those changes, a simple difference-in-differences. The lab below runs the calculation on the synthetic counts. Briefed stores rose more than comparison stores, suggesting the briefings helped, though the estimate rests on the assumption that both groups would otherwise have followed similar trends.
Lab — Difference-in-differences on report rates
# Difference-in-differences on report rates (reports per staff per month), synthetic.
briefed = {"before": [0.20, 0.25, 0.22], "after": [0.55, 0.58, 0.51]}
compare = {"before": [0.21, 0.19, 0.24], "after": [0.27, 0.25, 0.22]}
m = lambda xs: sum(xs) / len(xs)
db = m(briefed["after"]) - m(briefed["before"]); dc = m(compare["after"]) - m(compare["before"])
print(f"briefed change {db:+.2f}, comparison change {dc:+.2f}, difference-in-differences {db - dc:+.2f}")
Run with python3 lab_did.py. Verified output:
briefed change +0.32, comparison change +0.03, difference-in-differences +0.29
33.2 Threats to validity
Lucas uses the categories in Wohlin and colleagues (2012). Internal validity: briefed stores were chosen by regional managers, perhaps because they were already engaged. Construct validity: report-button use measures willingness to report, not ability to detect. External validity: Fernway's stores may differ from other retailers. Conclusion validity: with few stores, the result could arise by chance. He addresses some threats by checking that pre-briefing trends were similar, and he states the rest openly rather than hiding them in an appendix. Each threat is listed with its likely direction of bias.
33.3 Reporting with caution
Lucas writes that briefed stores showed a larger rise in reporting than comparison stores, consistent with a benefit from briefings, but that the design cannot rule out other explanations. He recommends a randomised rollout for the next round, where stores are assigned by lot, which Fernway agrees to try. Hannah uses the cautious result to keep briefings while the stronger study runs. Dr Bellini marks the report highly for matching the strength of its claims to the strength of its evidence.
Worked example
Lucas runs a difference-in-differences on synthetic report rates, lists four validity threats and reports a cautious finding with a randomised follow-up.
Synthetic case. Counts are synthetic; Wohlin et al. (2012) is real.
Errors and limits
Few units make estimates unstable. Parallel trends cannot be proven, only checked.
Exercises
- EX097 — DiD result. What was the difference-in-differences in the lab? Rubric: 3 = 0.29 reports per staff per month;; 2 = positive;; 1 = vague;; 0 = none.
- EX098 — Construct threat. What construct validity threat did Lucas name? Rubric: 3 = report-button use measures willingness, not detection ability;; 2 = construct;; 1 = vague;; 0 = none.
- EX099 — Stronger design. What did Lucas recommend next? Rubric: 3 = randomised rollout by lot;; 2 = randomise;; 1 = vague;; 0 = none.
Synthesis
Analysis should match claim strength to design strength and name validity threats openly.
Continue studying — chapter 33 of 40
← Chapter 32 (draft) · Course chapter index · Chapter 34 (draft) →