Chapter 5 — Main study and independent validation

Lesson map

Lesson 05 published. Map, example, EX027.

flowchart TD
  N0["Learning outcomes"]
  N1["5.1 Designing the main study"]
  N2["5.2 Effects on counts"]
  N3["5.3 Independent validation"]
  N4["5.4 Interpreting the findings"]
  N0 --> N1 --> N2 --> N3 --> N4
  N4 --> EX["Worked example"]
  EX --> Q["Exercise EX027"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["5.1 Designing the main study"]
  N2["5.2 Effects on counts"]
  N3["5.3 Independent validation"]
  N4["5.4 Interpreting the findings"]
  N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX027"]

[Learning outcomes]
    |
    v
[5.1 Designing the main study]
    |
    v
[5.2 Effects on counts]
    |
    v
[5.3 Independent validation]
    |
    v
[5.4 Interpreting the findings]
    |
    v
[Worked example] --> [Exercise EX027]

Learning outcomes

You will (1) run the main simulation of governance decisions; (2) measure effects on published counts; (3) validate results independently.

5.1 Designing the main study

The main study varies three decisions within ranges that Westmark staff described as plausible: the linkage threshold from 0 to 8, whether m and u are re-estimated each year, and whether blank sector codes are imputed or left blank. Each combination produces a synthetic set of regional employment counts. Nadia runs every combination with fixed random seeds and stores outputs with the script version. She compares each result with a reference produced by the true synthetic links, which are known because she generated the data herself.

5.2 Effects on counts

The results show that the threshold has the largest effect. At low thresholds, false links inflate employment in regions with many similarly named firms. At high thresholds, missed links deflate counts in regions where postcodes change often. Re-estimating m and u reduces both errors. Imputing sector codes shifts employment between sectors but barely changes regional totals. Nadia expresses each effect as the difference from the reference, so readers can see which decisions matter most and for which regions. Small regions show the largest relative errors.

5.3 Independent validation

To validate, Prof. Eriksen arranges for a researcher at another university, who has not seen Nadia's code, to implement the method from her written description and run it on a fresh synthetic replica generated with a different seed. The independent results agree in direction and approximate size for all three decisions. One discrepancy arises because Nadia's description omitted how ties at the threshold were treated. She corrects the description, showing how validation improves documentation as well as confidence in the results. Both runs are archived.

5.4 Interpreting the findings

Nadia relates the findings to governance. The decision with the largest effect, the threshold, is also the one with the least clear ownership. The decision with the smallest effect on regional totals, sector imputation, has the most documentation, because it appears in a published manual. She argues that governance attention at Westmark has followed visibility rather than impact. Prof. Eriksen warns her to avoid generalising from one organisation, and she frames the claim as a hypothesis for other statistics offices to test.

Worked example

Nadia varies threshold, re-estimation and imputation, finds the threshold has the largest effect on regional counts, has an independent researcher reproduce the direction and size of effects and links impact to unclear ownership.

Synthetic case. Results are synthetic.

Errors and limits

Plausible ranges come from interviews; other organisations may use wider ranges.

Exercises

• EX027 — Three decisions. Name the three decisions varied.
  Show marking criteria

  Rubric: 3 = threshold, m and u re-estimation, sector imputation;; 2 = all;; 1 = one;; 0 = none.
• EX028 — Largest effect. Which decision had the largest effect?
  Show marking criteria

  Rubric: 3 = the linkage threshold;; 2 = partly correct;; 1 = other;; 0 = none.
• EX029 — Low threshold. What happens to counts at low thresholds?
  Show marking criteria

  Rubric: 3 = false links inflate counts;; 2 = partly correct;; 1 = vague;; 0 = none.
• EX030 — Validation method. How was validation independent?
  Show marking criteria

  Rubric: 3 = another researcher reimplemented from the description with a new seed;; 2 = mentions reimplementation;; 1 = vague;; 0 = none.
• EX031 — Discrepancy. What caused the validation discrepancy?
  Show marking criteria

  Rubric: 3 = undocumented tie handling at the threshold;; 2 = partly correct;; 1 = vague;; 0 = none.
• EX032 — Attention. What has governance attention followed at Westmark?
  Show marking criteria

  Rubric: 3 = visibility rather than impact;; 2 = partly correct;; 1 = vague;; 0 = none.