Chapter 4 — Baseline replication and pilot
Lesson map
Lesson 04 published. Map, example, EX021.
flowchart TD
N0["Learning outcomes"]
N1["4.1 Why replicate first"]
N2["Lab — Granted versus used"]
N3["4.2 Reading the baseline"]
N4["4.3 Running the pilot"]
N5["4.4 Pilot findings"]
N6["4.5 Revising the procedure"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX021"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["4.1 Why replicate first"]
N2["Lab — Granted versus used"]
N3["4.2 Reading the baseline"]
N4["4.3 Running the pilot"]
N5["4.4 Pilot findings"]
N6["4.5 Revising the procedure"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX021"]
[Learning outcomes]
|
v
[4.1 Why replicate first]
|
v
[Lab — Granted versus used]
|
v
[4.2 Reading the baseline]
|
v
[4.3 Running the pilot]
|
v
[4.4 Pilot findings]
|
v
[4.5 Revising the procedure]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX021]
Learning outcomes
You will (1) replicate a baseline excess measurement on synthetic data; (2) run a pilot review; (3) revise the procedure from evidence.
4.1 Why replicate first
Before testing anything new, Rhea checks that she can reproduce a simple baseline: the share of granted entitlements not used in ninety days. Practitioner reports often cite such figures, but definitions vary. Replicating the measurement on synthetic data, then on one partner's export, confirms that her scripts, codebook and usage logs produce sensible results. If the baseline looked wildly different from what administrators expect, the problem would more likely lie in her pipeline than in the organisation, and finding that early is far cheaper than discovering it after the main study.
Lab — Granted versus used
"""Shadow comparison: entitlements granted versus used in 90 days for synthetic accounts (stdlib only)."""
granted = {"a01": {"ehr-read", "ehr-write", "rota"}, "a02": {"ehr-read", "billing"}, "a03": {"ehr-read", "ehr-write", "admin-console"},
"a04": {"rota"}, "a05": {"ehr-read", "billing", "rota", "payroll"}, "a06": {"ehr-read"}}
used = {"a01": {"ehr-read", "ehr-write", "rota"}, "a02": {"ehr-read"}, "a03": {"ehr-read", "ehr-write"},
"a04": {"rota"}, "a05": {"ehr-read", "rota"}, "a06": set()}
tot = unused = 0
for a in sorted(granted):
u = sorted(granted[a] - used[a]); tot += len(granted[a]); unused += len(u)
print(f"{a} granted={len(granted[a])} unused={u}")
print(f"total={tot} unused={unused} rate={unused/tot:.2f}")
Run with python3 lab_shadow.py. Verified output:
a01 granted=3 unused=[]
a02 granted=2 unused=['billing']
a03 granted=3 unused=['admin-console']
a04 granted=1 unused=[]
a05 granted=4 unused=['billing', 'payroll']
a06 granted=1 unused=['ehr-read']
total=14 unused=5 rate=0.36
4.2 Reading the baseline
In the synthetic sample, six accounts hold fourteen entitlements and five were not used in ninety days, an unused rate of 0.36. The pattern matters as much as the rate: account a05 holds payroll and billing without using either, suggesting accumulated access from earlier jobs; a03 holds an administration console unused; and a06 has used nothing, which might mean a leaver whose account was not disabled. Unused is not the same as excess: some access is needed rarely, such as year-end payroll. That is why Rhea's codebook asks coders to check documented purpose before calling an unused entitlement excess.
4.3 Running the pilot
The pilot applies the structured review in one small unit at Calder Valley with eleven reviewers and about six hundred entitlements. Rhea observes two sittings, notes where reviewers hesitate and records time per decision. She keeps the usual review running in a comparable unit for reference. The pilot is not meant to test the hypothesis; with one unit per condition it cannot. Its purpose is to find practical problems, such as confusing descriptions or missing usage data, while the cost of fixing them is still low and before any units are randomised.
4.4 Pilot findings
Three problems appear. Plain-language descriptions are missing for about a fifth of entitlements, so reviewers fall back on guessing. Usage data is absent for one legacy application that does not log access, so its entitlements always appear unused. And some reviewers manage people they have never met, because reporting lines in the directory are out of date. Rhea writes each finding with evidence, such as the count of missing descriptions, and agrees a fix with the partner: descriptions written by system owners, the legacy application marked as unknown rather than unused, and reporting lines corrected before the main study.
4.5 Revising the procedure
Each change to the procedure is recorded in a version history with date, reason and evidence. Rhea also checks that the changes do not favour the structured condition unfairly: correcting reporting lines helps both conditions, so it is applied to all units. Marking the legacy application as unknown changes the measure, so she recodes the baseline with the new rule and reports both versions. Separating pilot lessons from main-study results, and declaring every change, lets examiners see that the final design was improved by evidence rather than tuned to produce a desired outcome.
Worked example
The synthetic baseline shows 5 of 14 entitlements unused (rate 0.36); the pilot reveals missing descriptions, an unlogged legacy application and stale reporting lines, each fixed and recorded before the main study.
Synthetic case. Accounts, entitlements and the pilot unit are synthetic.
Errors and limits
Treating every unused entitlement as excess overstates the problem, because some legitimate access is used only occasionally.
Exercises
- EX021 — Unused count. How many entitlements are unused in the lab, out of how many? Rubric: 3 = 5 of 14, rate 0.36;; 2 = 5;; 1 = 14;; 0 = none.
- EX022 — Leaver signal. Which account might belong to a leaver, and why? Rubric: 3 = a06, used nothing in 90 days;; 2 = a06;; 1 = a01;; 0 = none.
- EX023 — Unused versus excess. Why is unused not the same as excess? Rubric: 3 = some access is needed rarely, such as year-end payroll;; 2 = rare use;; 1 = it is;; 0 = none.
- EX024 — Pilot purpose. What is the pilot for, if not testing the hypothesis? Rubric: 3 = finding practical problems while cheap to fix;; 2 = problems;; 1 = testing;; 0 = none.
- EX025 — Legacy fix. How is the unlogged legacy application handled? Rubric: 3 = marked unknown rather than unused, baseline recoded;; 2 = unknown;; 1 = removed;; 0 = none.
- EX026 — Fairness check. Why apply corrected reporting lines to all units? Rubric: 3 = it helps both conditions, so applying everywhere avoids bias;; 2 = fairness;; 1 = speed;; 0 = none.
Synthesis
Replicating a simple baseline and running a small pilot expose pipeline and procedure problems while they are still cheap to fix.
Continue studying — chapter 04 of 08
← Chapter 03 (draft) · Course chapter index · Chapter 05 (draft) →