Chapter 5 — Main study and independent validation
Lesson map
Lesson 05 published. Map, example, EX027.
flowchart TD
N0["Learning outcomes"]
N1["5.1 Running the study"]
N2["5.2 Calibration results"]
N3["Lab — Brier scores for matrix-band and calibrated fo"]
N4["5.3 Treatment order"]
N5["5.4 Independent validation"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX027"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["5.1 Running the study"]
N2["5.2 Calibration results"]
N3["Lab — Brier scores for matrix-band and calibrated fo"]
N4["5.3 Treatment order"]
N5["5.4 Independent validation"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX027"]
[Learning outcomes]
|
v
[5.1 Running the study]
|
v
[5.2 Calibration results]
|
v
[Lab — Brier scores for matrix-band and calibrated fo]
|
v
[5.3 Treatment order]
|
v
[5.4 Independent validation]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX027]
Learning outcomes
You will (1) run a randomised comparison of assessment methods; (2) analyse calibration with the Brier score; (3) validate results with an independent panel.
5.1 Running the study
Corvane's thirty assessors are randomly assigned to two groups of fifteen. One group uses the existing matrix with its band definitions; the other completes the ninety-minute calibration module and gives probability and loss ranges. Both groups assess the same twenty synthetic scenarios and forecast the probability of thirty short-horizon events that can be checked within the study, such as whether a monthly patching target will be missed. Amara monitors completion without seeing individual answers until the study closes, which protects assessors from pressure.
5.2 Calibration results
When the quarter ends, eleven of the thirty forecast events have occurred. The lab computes the Brier score for each group, where lower is better. The matrix group, whose forecasts are band midpoints, scores 0.180; the calibrated group scores 0.167. The difference favours the calibrated group but is modest, and with thirty events the uncertainty is wide. Amara reports the result as a small improvement consistent with her hypothesis, not as decisive proof, and notes that more events would be needed for a firm conclusion.
Lab — Brier scores for matrix-band and calibrated forecasts
# Calibration of assessors' probability forecasts for 30 synthetic quarterly events.
import random
r = random.Random(21)
events = [r.random() < p for p in [0.1] * 10 + [0.3] * 10 + [0.6] * 10]
true_p = [0.1] * 10 + [0.3] * 10 + [0.6] * 10
matrix_group = [0.5 if p >= 0.3 else 0.2 for p in true_p] # band midpoints from an ordinal scale
trained_group = [min(0.95, max(0.05, p + r.uniform(-0.08, 0.08))) for p in true_p]
def brier(f): return sum((fi - e) ** 2 for fi, e in zip(f, events)) / len(f)
print(f"events occurred {sum(events)}/30; Brier matrix-band group {brier(matrix_group):.3f}; Brier calibrated group {brier(trained_group):.3f}")
Run with python3 lab_brier.py. Verified output:
events occurred 11/30; Brier matrix-band group 0.180; Brier calibrated group 0.167
5.3 Treatment order
For decision quality, Amara compares each group's top five scenarios with the reference ranking by expected loss computed by an independent three-person panel. The calibrated group's top five contain four of the panel's top five; the matrix group's contain three, with two ties that had to be broken by discussion. She notes that ties forced the matrix group to rely on informal judgement, which is exactly where inconsistency enters. The difference is small in absolute terms but matters because treatment budgets follow the top of the list.
5.4 Independent validation
To validate the findings, Amara asks a second health board, also fictional, to repeat a shortened version of the study with twelve assessors. The direction of results is the same: slightly better calibration and closer treatment order for the calibrated group, though the differences are smaller. She treats this as partial replication. Independent validation guards against results that depend on one organisation's culture or on the enthusiasm of a single research partner, and it strengthens the claim she can make in her thesis.
Worked example
Amara randomises thirty assessors, finds Brier 0.167 versus 0.180, better top-five agreement for the calibrated group and partial replication elsewhere.
Synthetic case. Both boards and all data are synthetic.
Errors and limits
Thirty events give wide uncertainty. Assessors may share information across groups.
Exercises
- EX027 — Group allocation. How were assessors assigned to methods? Rubric: 3 = randomly, fifteen per group;; 2 = random;; 1 = by seniority;; 0 = none.
- EX028 — Events occurred. How many of thirty forecast events occurred? Rubric: 3 = eleven;; 2 = 11;; 1 = vague;; 0 = none.
- EX029 — Brier comparison. What Brier scores did the groups achieve? Rubric: 3 = matrix 0.180, calibrated 0.167;; 2 = calibrated lower;; 1 = vague;; 0 = none.
- EX030 — Top-five overlap. How many of the panel's top five did each group include? Rubric: 3 = calibrated four, matrix three;; 2 = four vs three;; 1 = vague;; 0 = none.
- EX031 — Tie effect. Why do ties matter in the matrix group? Rubric: 3 = they force informal judgement where inconsistency enters;; 2 = ties;; 1 = vague;; 0 = none.
- EX032 — Replication result. What did the second board's study show? Rubric: 3 = same direction, smaller differences;; 2 = partial replication;; 1 = vague;; 0 = none.
Synthesis
A randomised study with independent validation supports a modest, honestly stated advantage.
Continue studying — chapter 05 of 08
← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →