Chapter 6 — Data collection and analysis
Lesson map
Lesson 06 published. Map, example, EX033.
flowchart TD
N0["Learning outcomes"]
N1["6.1 Collecting the data"]
N2["6.2 The estimate"]
N3["6.3 Checking rivals"]
N4["6.4 What the estimate means"]
N5["Lab — Difference-in-differences of correction times"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX033"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["6.1 Collecting the data"]
N2["6.2 The estimate"]
N3["6.3 Checking rivals"]
N4["6.4 What the estimate means"]
N5["Lab — Difference-in-differences of correction times"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX033"]
[Learning outcomes]
|
v
[6.1 Collecting the data]
|
v
[6.2 The estimate]
|
v
[6.3 Checking rivals]
|
v
[6.4 What the estimate means]
|
v
[Lab — Difference-in-differences of correction times]
|
v
[Worked example] --> [Exercise EX033]
Learning outcomes
You will (1) collect department-level outcome data; (2) estimate a difference-in-differences effect with a bootstrap interval; (3) check rival explanations.
6.1 Collecting the data
Tomás extracts correction times for six months before and six months after the pilot, 120 corrections per group per period, using the definition from the pilot. Extraction follows a written script reviewed by the council's data team, which runs it inside the council's systems and returns only department-level tables. The analysis in this book reproduces those tables with synthetic data generated from the same summary figures, so every reader can run it without any access to council systems or residents' details. Nothing identifiable leaves.
6.2 The estimate
The lab shows mean correction times in stewarded departments falling from 24.1 to 14.8 days, while comparison departments move from 21.9 to 19.6 days. The difference-in-differences estimate is -7.0 days, with a bootstrap 95% interval from -9.1 to -4.8 days based on 2,000 resamples. The whole interval lies beyond the five-day threshold set in advance, so the pilot effect is both statistically clear and practically important by the criterion Tomás fixed before seeing data. The comparison departments also improved slightly, by about two days, perhaps because of the shared lookup tool, and the design subtracts that common change from the stewarded departments' gain.
6.3 Checking rivals
Error volumes are stable in both groups, so lower workload does not explain the change. Tenure figures show no shift of experienced staff. Monthly results show improvement throughout the six months rather than fading, which argues against a pure novelty effect. Within the stewarded group, both departments improve, including council tax, which had no prior informal sharing, weakening the culture rival. Tomás reports each check in a table alongside the main estimate, so readers can judge how well rivals were addressed.
6.4 What the estimate means
Tomás interprets the result carefully: in this synthetic reproduction of one council's pilot, appointing stewards with decision rights over address standards was associated with corrections reaching all systems about a week sooner. Interviews suggest the mechanism: stewards agreed a single address standard and a weekly propagation routine. He avoids claiming that stewardship works everywhere. Dr. Solberg notes that his careful wording respects both the evidence and the limits of a two-department pilot. Other councils may differ in staffing, systems and the number of departments holding addresses.
Lab — Difference-in-differences of correction times with bootstrap interval
"""Difference-in-differences of days to fix address errors, stewarded vs comparison departments, with bootstrap CI (stdlib only)."""
import random, statistics as st
random.seed(30)
def sample(mu, n): return [max(1, random.gauss(mu, 6)) for _ in range(n)]
treat_pre, treat_post = sample(24, 120), sample(15, 120)
comp_pre, comp_post = sample(22, 120), sample(19, 120)
def did(a, b, c, d): return (st.mean(b) - st.mean(a)) - (st.mean(d) - st.mean(c))
est = did(treat_pre, treat_post, comp_pre, comp_post)
boots = sorted(did(*(random.choices(x, k=len(x)) for x in (treat_pre, treat_post, comp_pre, comp_post))) for _ in range(2000))
print(f"mean days: stewarded {st.mean(treat_pre):.1f} -> {st.mean(treat_post):.1f}; comparison {st.mean(comp_pre):.1f} -> {st.mean(comp_post):.1f}")
print(f"difference-in-differences {est:.1f} days (95% CI {boots[50]:.1f} to {boots[1949]:.1f})")
Run with python3 lab_did.py. Verified output:
mean days: stewarded 24.1 -> 14.8; comparison 21.9 -> 19.6
difference-in-differences -7.0 days (95% CI -9.1 to -4.8)
Worked example
Tomás estimates a difference-in-differences of -7.0 days (95% CI -9.1 to -4.8), beyond the five-day threshold, checks volume, tenure, novelty and culture rivals and identifies a single standard and weekly propagation as the mechanism.
Synthetic case. Data are synthetic.
Errors and limits
A bootstrap interval reflects sampling variation, not design bias.
Exercises
- EX033 — Stewarded change. How did stewarded correction times change? Rubric: 3 = 24.1 to 14.8 days;; 2 = partly correct;; 1 = wrong;; 0 = none.
- EX034 — DiD estimate. What was the difference-in-differences estimate? Rubric: 3 = -7.0 days;; 2 = partly correct;; 1 = wrong;; 0 = none.
- EX035 — Interval. What was the 95% interval? Rubric: 3 = -9.1 to -4.8 days;; 2 = partly correct;; 1 = wrong;; 0 = none.
- EX036 — Threshold met. Does the interval exceed the five-day threshold? Rubric: 3 = yes, entirely;; 2 = partly correct;; 1 = no;; 0 = none.
- EX037 — Culture rival. Why is the culture rival weakened? Rubric: 3 = council tax improved without prior sharing;; 2 = partly correct;; 1 = vague;; 0 = none.
- EX038 — Mechanism. What mechanism did interviews suggest? Rubric: 3 = single address standard and weekly propagation;; 2 = both;; 1 = one;; 0 = none.
Continue studying — lesson 06 (chapter 06 of 08)
← Chapter 05 · Course chapter index · Chapter 07 →
Continue to the next authored lesson.