Chapter 5 — Main study and independent validation
Lesson map
Lesson 05 published. Map, example, EX027.
flowchart TD
N0["Learning outcomes"]
N1["5.1 Running the study"]
N2["5.2 Difference-in-differences"]
N3["Lab — Difference-in-differences with permutation che"]
N4["5.3 Reading the estimate"]
N5["5.4 Independent validation"]
N6["5.5 Reporting uncertainty"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX027"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["5.1 Running the study"]
N2["5.2 Difference-in-differences"]
N3["Lab — Difference-in-differences with permutation che"]
N4["5.3 Reading the estimate"]
N5["5.4 Independent validation"]
N6["5.5 Reporting uncertainty"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX027"]
[Learning outcomes]
|
v
[5.1 Running the study]
|
v
[5.2 Difference-in-differences]
|
v
[Lab — Difference-in-differences with permutation che]
|
v
[5.3 Reading the estimate]
|
v
[5.4 Independent validation]
|
v
[5.5 Reporting uncertainty]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX027]
Learning outcomes
You will (1) run a staggered adoption study with comparison teams; (2) estimate an effect with difference-in-differences; (3) obtain independent validation of coding and analysis; (4) report uncertainty honestly.
5.1 Running the study
Over eighteen months, eighteen Tessaro teams adopt the protocol in three waves as budgets allow, while twelve teams continue design review without it. Amara collects design changes, sessions, fidelity scores and defects monthly. Concurrent changes, such as one division introducing security champions, are recorded in a quarterly log. She avoids influencing outcomes: she does not attend sessions as a participant, and the coding assistant who classifies defects does not know which teams adopted. Blinding the coding protects the outcome measure from the very natural wish to see the intervention succeed.
5.2 Difference-in-differences
The analysis compares the change in defect rate for adopting teams with the change for comparison teams over the same period. If both groups improve equally, the difference is zero and the practice has no measurable effect beyond the general trend. The method assumes parallel trends: without the intervention, both groups would have moved similarly. Amara checks this assumption on pre-adoption data and finds trends close enough to proceed. The lab below computes the estimate on synthetic team-period rates and a simple permutation check, which shuffles adoption labels to see how often a difference as large arises by chance.
Lab — Difference-in-differences with permutation check
# Difference-in-differences on synthetic team rates (defects per 100 design changes) + permutation check.
import random, statistics
random.seed(5)
adopt = [(round(random.gauss(3.9, 0.5), 2), None) for _ in range(18)]
comp = [(round(random.gauss(3.7, 0.5), 2), None) for _ in range(12)]
adopt = [(b, round(b - 1.1 + random.gauss(0, 0.9), 2)) for b, _ in adopt]
comp = [(b, round(b - 0.3 + random.gauss(0, 0.9), 2)) for b, _ in comp]
def did(a, c):
return statistics.mean(y - x for x, y in a) - statistics.mean(y - x for x, y in c)
est = did(adopt, comp)
print(f"adopting: {statistics.mean(x for x,_ in adopt):.2f} -> {statistics.mean(y for _,y in adopt):.2f}")
print(f"comparison: {statistics.mean(x for x,_ in comp):.2f} -> {statistics.mean(y for _,y in comp):.2f}")
print(f"DiD estimate: {est:.2f} per 100 changes")
allt = adopt + comp; hits = 0; R = 5000
for _ in range(R):
random.shuffle(allt)
if did(allt[:18], allt[18:]) <= est: hits += 1
print(f"permutation share as extreme: {hits / R:.3f}")
Run with python3 lab_did.py. Verified output:
adopting: 3.75 -> 2.80
comparison: 3.83 -> 3.65
DiD estimate: -0.77 per 100 changes
permutation share as extreme: 0.022
5.3 Reading the estimate
On the synthetic data, adopting teams fell from 3.75 to 2.80 defects per hundred changes (a change of -0.95) while comparison teams fell from 3.83 to 3.65 (-0.18). The difference-in-differences is -0.77, about minus 0.8, roughly a fifth fewer defects relative to baseline. The permutation check shows a difference this large in about two per cent of shuffles. Amara reports the estimate with that uncertainty and notes it is slightly smaller than the quarter Tessaro hoped for. She avoids the word proves and writes that the data are consistent with a modest reduction under the stated assumptions.
5.4 Independent validation
A second researcher at another university, given only the coding manual and a random sample of defects, recodes them blind and reaches substantial agreement with the original coding. A statistician independently reruns the analysis from Amara's archived scripts and obtains the same estimate. Validation also covers a sensitivity analysis: excluding the division that introduced security champions changes the estimate only slightly, which weakens that rival explanation. Each validation step is reported with who did it, what they received and what they found, including one small coding disagreement that was resolved by clarifying a definition.
5.5 Reporting uncertainty
Amara's results section leads with the effect estimate and its interval, then fidelity data, then sensitivity analyses, then limitations. She reports that teams with low fidelity scores showed little change, which is consistent with the practice mattering but could also reflect less engaged teams. She avoids recommending universal adoption, instead stating that at Tessaro the practice was associated with a modest reduction at a cost of about ninety minutes per change. Honest uncertainty makes the result useful to other organisations deciding whether to try the practice under their own conditions.
Worked example
Amara runs the eighteen-month study, estimates a difference-in-differences of about minus 0.8 defects per hundred changes with a permutation p of about 0.02, obtains blind recoding and an independent rerun, and reports fidelity and sensitivity results.
Synthetic case. All team data, estimates and validators are synthetic.
Errors and limits
Parallel trends can hold before adoption yet fail afterwards. Permutation checks with few teams have coarse resolution.
Exercises
- EX027 — Blind coding. Why does the coding assistant not know which teams adopted? Rubric: 3 = to protect the outcome measure from expectation bias;; 2 = bias;; 1 = speed;; 0 = none.
- EX028 — Parallel trends. What assumption does difference-in-differences need? Rubric: 3 = parallel trends between groups without the intervention;; 2 = named;; 1 = vague;; 0 = none.
- EX029 — Lab estimate. What difference-in-differences and permutation result did the lab print? Rubric: 3 = about -0.77 per hundred changes, permutation share about 0.02;; 2 = estimate only;; 1 = positive;; 0 = none.
- EX030 — Champions sensitivity. What did excluding the security-champions division show? Rubric: 3 = estimate changed only slightly, weakening that rival;; 2 = slight change;; 1 = large change;; 0 = none.
- EX031 — Low fidelity reading. Why is the low-fidelity finding ambiguous? Rubric: 3 = consistent with the practice mattering or with less engaged teams;; 2 = both readings;; 1 = one reading;; 0 = none.
- EX032 — Avoided claim. What recommendation does Amara avoid, and what does she state instead? Rubric: 3 = avoids universal adoption; states modest reduction at Tessaro at about ninety minutes per change;; 2 = partial;; 1 = recommends universal;; 0 = none.
Synthesis
A main study earns credibility through comparison groups, blind measurement, independent validation and plainly reported uncertainty.
Continue studying — chapter 05 of 08
← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →