Chapter 4 — Baseline replication and pilot
Lesson map
Lesson 04 published. Map, example, EX021.
flowchart TD
N0["Learning outcomes"]
N1["4.1 Why replicate first"]
N2["4.2 The pilot lab"]
N3["4.3 Who chooses the threshold"]
N4["4.4 Pilot lessons"]
N5["Lab — Match weights, thresholds, precision and recal"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX021"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["4.1 Why replicate first"]
N2["4.2 The pilot lab"]
N3["4.3 Who chooses the threshold"]
N4["4.4 Pilot lessons"]
N5["Lab — Match weights, thresholds, precision and recal"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX021"]
[Learning outcomes]
|
v
[4.1 Why replicate first]
|
v
[4.2 The pilot lab]
|
v
[4.3 Who chooses the threshold]
|
v
[4.4 Pilot lessons]
|
v
[Lab — Match weights, thresholds, precision and recal]
|
v
[Worked example] --> [Exercise EX021]
Learning outcomes
You will (1) replicate a record linkage baseline; (2) compute match weights and thresholds; (3) report the trade-off between precision and recall.
4.1 Why replicate first
Before testing her own ideas, Nadia replicates Westmark's published linkage method on the synthetic replica. Westmark uses a Fellegi and Sunter approach: each field contributes a weight based on m, the probability that a true match agrees on the field, and u, the probability that a non-match agrees by chance. Agreement adds the logarithm of m over u; disagreement adds the logarithm of one minus m over one minus u. Replicating the baseline proves that her tools reproduce the method before she varies the decisions within it.
4.2 The pilot lab
The pilot uses 4,000 synthetic record pairs, 400 of them true matches, with name, postcode and sector fields. At a threshold of 0, it links 505 pairs with precision 0.776 and recall 0.980. At a threshold of 4, it links 386 pairs with precision 0.953 and recall 0.920. At a threshold of 8, it links 338 pairs with precision 0.991 and recall 0.838. Raising the threshold removes false matches but loses true ones, so no threshold is free of error. Between thresholds 0 and 4, precision gains far more than recall loses, which suggests the lowest threshold links too many false pairs. Beyond 4, each gain in precision costs noticeably more recall.
4.3 Who chooses the threshold
Nadia asks Westmark staff who chose the current threshold. The answer is unclear: a methodologist set it years ago, and later releases reused it without review. Interviews suggest that the production team believes the methods team owns it, while the methods team believes production can adjust it. This is exactly the decision-rights gap her proposal anticipated. She records the finding in her decision matrix and notes that the threshold affects every regional employment count Westmark publishes. Neither team had a written mandate covering the decision.
4.4 Pilot lessons
The pilot shows that the replica and scripts work, that the precision and recall trade-off is substantial and that ownership of the threshold is unclear. It also reveals that m and u values were estimated once and never updated, although the register has changed. Prof. Eriksen asks Nadia to treat the m and u estimates as another governance decision in the main study. She updates her pre-registration log to note the addition before running the main study, keeping the change transparent. Transparency comes first.
Lab — Match weights, thresholds, precision and recall
"""Fellegi-Sunter style match weights on synthetic record pairs; precision and recall by threshold (stdlib only)."""
import random, math
random.seed(9)
fields = {"name": (0.95, 0.01), "postcode": (0.90, 0.05), "sector": (0.85, 0.20)} # (m, u)
pairs = []
for i in range(4000):
match = i < 400
agree = {f: (random.random() < (m if match else u)) for f, (m, u) in fields.items()}
w = sum(math.log2(m / u) if agree[f] else math.log2((1 - m) / (1 - u)) for f, (m, u) in fields.items())
pairs.append((w, match))
for t in (0, 4, 8):
pred = [p for p in pairs if p[0] >= t]; tp = sum(1 for p in pred if p[1])
print(f"threshold {t:2d}: linked {len(pred):4d} precision {tp/len(pred):.3f} recall {tp/400:.3f}")
Run with python3 lab_linkage.py. Verified output:
threshold 0: linked 505 precision 0.776 recall 0.980
threshold 4: linked 386 precision 0.953 recall 0.920
threshold 8: linked 338 precision 0.991 recall 0.838
Worked example
Nadia replicates Fellegi-Sunter weights on 4,000 synthetic pairs, reports precision from 0.776 to 0.991 and recall from 0.980 to 0.838 across thresholds and finds unclear ownership of the threshold.
Synthetic case. Pairs are synthetic; Fellegi and Sunter (1969) is real.
Errors and limits
Assuming field independence, as this pilot does, can distort weights; test it later.
Exercises
- EX021 — Weight formula. What does agreement on a field add to the weight? Rubric: 3 = log of m over u;; 2 = partly correct;; 1 = wrong;; 0 = none.
- EX022 — Threshold 4. What precision and recall occur at threshold 4? Rubric: 3 = 0.953 and 0.920;; 2 = both;; 1 = one;; 0 = none.
- EX023 — Higher threshold. What happens when the threshold rises to 8? Rubric: 3 = fewer false matches, more missed true ones;; 2 = mentions trade-off;; 1 = vague;; 0 = none.
- EX024 — Unclear owner. Who owns the threshold at Westmark? Rubric: 3 = unclear; teams point to each other;; 2 = mentions unclear;; 1 = names one;; 0 = none.
- EX025 — Stale estimates. What else had never been updated? Rubric: 3 = m and u estimates;; 2 = partly correct;; 1 = vague;; 0 = none.
- EX026 — Log update. What did Nadia do before adding the new variable? Rubric: 3 = updated her pre-registration log;; 2 = partly correct;; 1 = vague;; 0 = none.
Continue studying — lesson 04 (chapter 04 of 08)
← Chapter 03 · Course chapter index · Chapter 05 →
Continue to the next authored lesson.