Chapter 5 — Proposal and pilot
Lesson map
Lesson 05 published. Map, example, EX027.
flowchart TD
N0["Learning outcomes"]
N1["5.1 Proposal"]
N2["5.2 Pilot extract"]
N3["Lab — pilot completeness"]
N4["5.3 Reading completeness"]
N5["5.4 Fixing gaps"]
N6["5.5 Revising the plan"]
N7["Pipeline checks"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX027"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["5.1 Proposal"]
N2["5.2 Pilot extract"]
N3["Lab — pilot completeness"]
N4["5.3 Reading completeness"]
N5["5.4 Fixing gaps"]
N6["5.5 Revising the plan"]
N7["Pipeline checks"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX027"]
[Learning outcomes]
|
v
[5.1 Proposal]
|
v
[5.2 Pilot extract]
|
v
[Lab — pilot completeness]
|
v
[5.3 Reading completeness]
|
v
[5.4 Fixing gaps]
|
v
[5.5 Revising the plan]
|
v
[Pipeline checks]
|
v
[Worked example] --> [Exercise EX027]
Learning outcomes
You will (1) write a proposal with analysis plan; (2) pilot the data extract; (3) check completeness before analysis.
5.1 Proposal
The proposal sets out the question, three explanations with predictions, data, ethics terms and analysis: compare drift proportions across groups with risk ratios and chi-square tests, then stratify by department.
The proposal also commits to how results will be read: a risk ratio above one with a chi-square above 3.84 is treated as a clear association at the 5% level with one degree of freedom, and anything below as unclear. Fixing this threshold in advance prevents a marginal result, such as engineering's later chi-square equal to 2.1, being described as a finding.
5.2 Pilot extract
Halvorsen's IT team provides a pilot extract of 120 laptops for one month to test the pipeline.
One month for 120 laptops is small enough to inspect by hand and large enough to show the problems a full extract would have: missing fields, inconsistent department names and dates in different formats. Testing the pipeline here means the six-month, 600-laptop extract can be analysed as planned rather than repaired under deadline. The pilot extract also contains the drift outcome for the month, so Chloé can confirm the script computes drift exactly as the seminar lab does on the six-laptop sample before trusting it on 600.
Lab — pilot completeness
"""Check completeness of a synthetic pilot extract before analysis (stdlib only)."""
rows = 120
missing = {"last_policy_refresh": 9, "local_admin_flag": 0, "location": 4, "department": 2}
for k, m in missing.items():
print(f"{k}: {rows-m}/{rows} present ({(rows-m)/rows*100:.1f}%)")
complete = rows - 13
print(f"rows complete on all fields: {complete}/{rows} ({complete/rows*100:.1f}%)")
Run with python3 lab_pilot.py. Verified output:
last_policy_refresh: 111/120 present (92.5%)
local_admin_flag: 120/120 present (100.0%)
location: 116/120 present (96.7%)
department: 118/120 present (98.3%)
rows complete on all fields: 107/120 (89.2%)
5.3 Reading completeness
Local admin flag is complete (120/120). Last policy refresh is present for 111 (92.5%), location for 116 (96.7%) and department for 118 (98.3%). Some rows miss more than one field; 107 of 120 rows (89.2%) are complete on every field.
The missing fields add up to 15 gaps, but only 13 rows are incomplete, because some rows miss more than one field. The local admin flag, which Explanation A depends on, is complete. The refresh date, which Explanation B depends on, is the field most often missing, which is the first warning that dropping incomplete rows would bias the test of B.
5.4 Fixing gaps
Missing refresh dates come from laptops that had not checked in during the month. Rather than drop them, Chloé records them as "no refresh in month", which is itself relevant to Explanation B.
Sven's question, whether missing data bias the results, is answered by this choice. Dropping the nine laptops without a refresh date would remove exactly the devices that Explanation B predicts will drift most, and so make remote working look less important than it might be. Coding them as "no refresh in month" keeps them in the analysis as an extreme value of the very variable B is about.
5.5 Revising the plan
The plan now states how each missing field is handled.
For the refresh date, missing becomes "no refresh in month"; for location and department, missing rows are reported as a separate category and excluded only from the comparison that needs that field. Writing these rules into the plan before the main extract arrives means the handling cannot be adjusted after seeing which rule gives the clearest result. Dr Mehta signs the revised plan with its date, before the six-month extract is requested from Halvorsen's IT team.
Pipeline checks
Beyond completeness, Chloé checks that device IDs are unique, that dates fall within the month and that setting values match expected formats. Two devices appear twice because they were renamed mid-month; she merges them using the pseudonymised hardware ID. These checks take a day but prevent errors that would be hard to find later.
Pilot is not results
She does not look at drift rates by group in the pilot, so the pilot cannot influence her predictions.
Supervisor review
Dr Mehta reviews the pilot report and confirms the missing-data rules before the full extract is requested.
Timeline
The full extract follows two weeks later.
Version
The plan is versioned.
Worked example
Sven asks whether missing data biases results. Chloé shows that dropping no-refresh laptops would hide exactly the devices Explanation B is about.
Synthetic case. Synthetic.
Errors and limits
Dropping incomplete rows silently can remove the cases that matter most.
Exercises
- EX027 — Analysis plan. Which tests does the plan use? Rubric: 3 = risk ratios, chi-square, stratification; 2 = two; 1 = one; 0 = none.
- EX028 — Pilot size. How large was the pilot extract? Rubric: 3 = 120 laptops one month; 2 = partial; 1 = vague; 0 = none.
- EX029 — Lab figures. How many rows were complete on all fields? Rubric: 3 = 107/120 (89.2%); 2 = partial; 1 = vague; 0 = none.
- EX030 — Refresh gaps. Why were refresh dates missing? Rubric: 3 = laptops did not check in; 2 = partial; 1 = vague; 0 = none.
- EX031 — Handling. How were missing refresh dates recorded? Rubric: 3 = no refresh in month; 2 = partial; 1 = vague; 0 = none.
- EX032 — Bias. Why not drop them? Rubric: 3 = they are the cases for Explanation B; 2 = partial; 1 = vague; 0 = none.
Synthesis
Pilot the data and decide missing-data rules before analysis.
Continue studying — chapter 05 of 08
← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →