Chapter 5 — Proposal and pilot

Chapter 5 — Proposal and pilot

Lesson map

Lesson 05 published. Map, example, EX027.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["5.1 The proposal as a contract"]
  N2["5.2 Timeline and critical path"]
  N3["5.3 The risk register"]
  N4["5.4 Asking custodians for data"]
  N5["5.5 What a pilot is for"]
  N6["5.6 Progression criteria"]
  N7["5.7 Practical activity: auditing the pilot extract"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX027"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["5.1 The proposal as a contract"]
  N2["5.2 Timeline and critical path"]
  N3["5.3 The risk register"]
  N4["5.4 Asking custodians for data"]
  N5["5.5 What a pilot is for"]
  N6["5.6 Progression criteria"]
  N7["5.7 Practical activity: auditing the pilot extract"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX027"]

[Learning outcomes and connection]
    |
    v
[5.1 The proposal as a contract]
    |
    v
[5.2 Timeline and critical path]
    |
    v
[5.3 The risk register]
    |
    v
[5.4 Asking custodians for data]
    |
    v
[5.5 What a pilot is for]
    |
    v
[5.6 Progression criteria]
    |
    v
[5.7 Practical activity: auditing the pilot extract]
    |
    v
[Worked example] --> [Exercise EX027]

Learning outcomes and connection

By the end of this chapter the learner will be able to:

1. Write a research proposal that functions as a contract with a supervisor, a committee and data partners: specific enough to be checked, flexible only where flexibility is declared.
2. Plan a two-year master's project with its critical path, and keep a risk register with triggers and owners.
3. Approach data custodians with a precise request and negotiate access that the study can survive.
4. Design a feasibility pilot whose purpose is to test procedures, not to estimate effects, and explain why pilot effect estimates should not be used to justify the main study.
5. Set predeclared progression (go/no-go) criteria, including an amend-and-re-pilot outcome.
6. Audit a pilot data extract systematically and turn each finding into a question for the custodian or a change in procedure.

Chapters 1–4 produced the question, the seminar map, the design and the rival table. This chapter packages them into a proposal and tests whether the study can actually be done. Chapter 6 runs it.

5.1 The proposal as a contract

A proposal answers three audiences at once. The supervisor needs to know the project is feasible and the candidate understands it. The committee (and later the examiners) needs to see that the question, design and analysis are coherent. Data partners need to know exactly what will be requested, how it will be protected and what will be published. A proposal that satisfies all three is specific about what is fixed and explicit about what is open.

A standard structure for the Spillover Study:

1. Title and summary (250 words): question, design, data, expected contribution.
2. Problem and rationale: the decision that the answer informs; the gap from the concept matrix (Chapter 1).
3. Questions and estimands: RQ1–RQ3 with the estimand definitions.
4. Background: a two-page concept-centric summary of the literature.
5. Design: controlled interrupted time series and interviews; why these and not others.
6. Data and access: custodians, variables, period, aggregation, agreements, fallback sources.
7. Ethics and information protection: the approved or submitted application; sensitivity tiers.
8. Analysis plan: a summary of the pre-analysis plan with a link to its registration.
9. Rivals and robustness: the Chapter 4 rival table with tests.
10. Timeline and milestones.
11. Risks and mitigations.
12. Outputs: dissertation, partner briefing, possible paper; what will and will not be published.
13. References.

Mark each element as fixed (changes require a dated entry in the deviation log and the supervisor's agreement) or open (to be decided by a stated date or after the pilot). In the Spillover Study, the estimands, primary outcome, comparison region and model are fixed; the exact number of interviews (8–12) and the secondary outcome's definition (pending the custodian's data dictionary) are open.

5.2 Timeline and critical path

A two-year master's at about 1,200 hours of study time leaves perhaps 600–700 hours for the dissertation itself once taught components are counted; the rest of this course's hours are the chapters you are studying. The scarce resource is not analysis time but elapsed time waiting for other people: ethics committees, data custodians, partners' legal teams and interviewees' calendars.

The critical path is the chain of tasks whose delay delays the whole project: here, ethics approval and data agreements, which must both precede the pilot. Start them early, even before the design is final, by sharing a draft data request and asking custodians what approvals they require.

5.3 The risk register

Review the register at every supervision meeting. A risk register that is written once for the proposal and never opened again is decoration.

5.4 Asking custodians for data

Custodians receive many vague requests and answer precise ones. A good request states: the purpose in two sentences; the exact variables and their definitions as you understand them; the period; the aggregation (daily counts per ED; no patient-level records); how the data will be stored and destroyed; what will be published and in what form; the ethics approval status; and what you will give back (a briefing, a check of their data's quality). Ask the custodian three questions that every time-series study needs: Did the recording system, its definitions or the site's catchment change during the period? Are there known gaps or downtime days? How are late entries and corrections handled? The answers often matter more than the data.

For the Spillover Study, Alder Point and Brookfield agreed to supply daily aggregates; Calloway agreed after the coalition intervened; Wexmoor's counts came from a public regional dashboard whose terms allow research use, which the candidate recorded in the data management plan.

5.5 What a pilot is for

A pilot or feasibility study asks whether the main study can be done as planned: whether data can be obtained in the expected form, whether participants can be recruited, whether the procedures and instruments work and how long they take. Thabane et al. (2010) give a tutorial on the what, why and how; Lancaster, Dodd and Williamson (2004) recommend that pilots have clear feasibility objectives and be analysed descriptively; the CONSORT extension for pilot and feasibility trials (Eldridge et al., 2016) asks authors to report the criteria used to decide whether to proceed. Although those sources address trials, their core logic transfers directly to observational and mixed-methods studies.

What a pilot is not for: estimating the effect, or estimating the effect size for a power calculation. Leon, Davis and Kraemer (2011) explain why: pilot samples are small, so their effect estimates are too imprecise to guide sample-size decisions and can mislead in either direction. In an observational event study there is an additional reason: if the candidate examines outcome data around the event during the "pilot", the subsequent analysis is no longer planned blind to the result. The Spillover Study's data pilot therefore uses a 35-day baseline extract from Alder Point, entirely before day −60, and examines completeness and plausibility only.

The Spillover Study's pilot has three components:

1. Data pilot: the baseline extract, audited against predeclared criteria (Lab 5).
2. Interview pilot: two interviews with people outside the sample (a retired ED manager and an EMS educator from another region), testing the guide's clarity, length and sensitivity.
3. Pipeline pilot: the full analysis code run end to end on synthetic data (Lab 4's generator) to check that it produces every planned table and figure.

5.6 Progression criteria

Progression criteria state in advance what pilot results lead to proceeding, amending or stopping. A common format uses three zones: go (proceed as planned), amend (proceed after specified changes, possibly re-piloting) and stop (the study as designed is not feasible). The Spillover Study's data criteria:

Interview criteria: the guide fits in 45 minutes; neither pilot participant reports discomfort with any question; both understand every question without rephrasing. Pipeline criterion: every planned output produced without manual steps.

5.7 Practical activity: auditing the pilot extract

The lab below audits a synthetic 35-day extract from Alder Point against the criteria in §5.6. It checks site codes, duplicate dates (and whether duplicates conflict), missing values, out-of-range values, missing calendar days and implausibly low days, and reports a decision.

# Lab 5 - audit a pilot extract against predeclared go/no-go criteria (offline, standard library only)
# A synthetic 35-day daily extract from one neighbouring hospital (AP), as a partner might send it.
import csv, io, datetime as dt

EXTRACT = """date,site,amb_arrivals,walkins,diversion_min,median_wait_min
2025-03-01,AP,41,96,0,27
2025-03-02,AP,37,88,0,25
2025-03-03,AP,44,102,35,31
2025-03-04,AP,40,99,0,29
2025-03-05,AP,39,,0,28
2025-03-06,AP,42,101,0,30
2025-03-07,AP,38,93,0,26
2025-03-08,AP,36,90,0,24
2025-03-09,AP,35,87,0,23
2025-03-10,AP,45,104,120,33
2025-03-11,AP,43,100,0,30
2025-03-11,AP,43,100,0,30
2025-03-12,AP,41,97,0,N/A
2025-03-13,AP,40,95,0,29
2025-03-15,AP,37,91,0,25
2025-03-16,AP,36,89,0,24
2025-03-17,AP,46,105,-15,34
2025-03-18,AP,44,103,0,31
2025-03-19,ALDR,42,98,0,29
2025-03-20,AP,41,96,0,28
2025-03-21,AP,39,94,0,27
2025-03-22,AP,37,90,0,26
2025-03-23,AP,35,86,0,24
2025-03-24,AP,47,106,1600,35
2025-03-25,AP,44,101,0,31
2025-03-26,AP,43,99,0,30
2025-03-27,AP,41,97,0,29
2025-03-28,AP,40,95,0,28
2025-03-29,AP,38,92,0,26
2025-03-30,AP,23,55,0,25
2025-03-31,AP,45,103,0,32
2025-04-01,AP,44,102,0,31
2025-04-02,AP,42,99,0,30
2025-04-03,AP,40,96,0,29
2025-04-04,AP,39,94,0,28
"""
FIELDS = ["amb_arrivals", "walkins", "diversion_min", "median_wait_min"]
rows = list(csv.DictReader(io.StringIO(EXTRACT)))
issues, seen, by_date = [], {}, {}
for i, r in enumerate(rows, start=2):                      # line numbers as in the file
    d = dt.date.fromisoformat(r["date"])
    if r["site"] != "AP":
        issues.append((i, "site code", r["site"]))
    if d in seen:
        same = all(r[f] == rows[seen[d] - 2][f] for f in FIELDS)
        issues.append((i, "duplicate date", "identical" if same else "CONFLICTING"))
        continue
    seen[d] = i
    by_date[d] = r
    for f in FIELDS:
        v = r[f].strip()
        if v in ("", "N/A", "NA"):
            issues.append((i, "missing", f))
            continue
        x = float(v)
        if x < 0 or (f == "diversion_min" and x > 1440):
            issues.append((i, "out of range", f"{f}={v}"))

start, end = min(by_date), max(by_date)
expected = (end - start).days + 1
missing_days = [start + dt.timedelta(k) for k in range(expected) if start + dt.timedelta(k) not in by_date]
cells = expected * len(FIELDS)
bad_cells = sum(1 for _, kind, _ in issues if kind in ("missing", "out of range")) + len(missing_days) * len(FIELDS)
arr = [int(by_date[d]["amb_arrivals"]) for d in sorted(by_date)]
mu = sum(arr) / len(arr)
low = [(str(d), int(r["amb_arrivals"])) for d, r in sorted(by_date.items()) if int(r["amb_arrivals"]) < 0.7 * mu]

for line, kind, detail in issues:
    print(f"line {line:2d}: {kind}: {detail}")
print("missing calendar days:", [str(d) for d in missing_days])
print("implausibly low days (<70% of mean):", low)
complete = 1 - bad_cells / cells
print(f"cell completeness {complete:.3f} over {expected} days x {len(FIELDS)} fields")
CRITERIA = {                                                # predeclared in the pilot protocol
    "completeness >= 0.97": complete >= 0.97,
    "no conflicting duplicates": not any(d == "CONFLICTING" for _, k, d in issues if k == "duplicate date"),
    "<= 1 missing calendar day": len(missing_days) <= 1,
    "all values in range": not any(k == "out of range" for _, k, _ in issues),
}
for c, ok in CRITERIA.items():
    print(f"  {'PASS' if ok else 'FAIL'}  {c}")
print("decision:", "GO" if all(CRITERIA.values()) else "AMEND AND RE-PILOT (no outcome analysis)")

Expected output:

line  6: missing: walkins
line 13: duplicate date: identical
line 14: missing: median_wait_min
line 18: out of range: diversion_min=-15
line 20: site code: ALDR
line 25: out of range: diversion_min=1600
missing calendar days: ['2025-03-14']
implausibly low days (<70% of mean): [('2025-03-30', 23)]
cell completeness 0.943 over 35 days x 4 fields
  FAIL  completeness >= 0.97
  PASS  no conflicting duplicates
  PASS  <= 1 missing calendar day
  FAIL  all values in range
decision: AMEND AND RE-PILOT (no outcome analysis)

Each finding becomes a question, not a silent fix:

• Line 13, duplicate date (identical): likely an export artefact. Ask how the extract is produced; remove the duplicate only after the custodian confirms.
• Lines 6 and 14, missing values: ask whether walk-ins on 5 March and waiting time on 12 March were not recorded or not exported. Do not impute in the pilot.
• Line 18, diversion −15 minutes; line 25, 1,600 minutes: impossible values. A negative duration suggests a sign or subtraction error; 1,600 minutes exceeds a day. Ask the custodian for the source records and how diversion minutes are computed.
• Line 20, site code ALDR: a legacy or alternative code for Alder Point. Ask for the code list; if ALDR is another site, the row must be removed.
• 14 March missing: ask whether the day was lost in export or not recorded.
• 30 March, 23 arrivals (about 57% of the mean): not impossible, but implausible. In the synthetic scenario the custodian explains that Alder Point had a six-hour electronic health record downtime that day; arrivals were logged on paper and only partly entered afterwards. That is a measurement artefact at a neighbouring hospital — the very thing a study of IT disruption must expect. The main study adds a rule: days flagged by the custodian as downtime days are identified in the codebook and analysed with and without them.

Completeness is 94.3%, in the amend zone, and two values are out of range. The decision is to amend and re-pilot — and, as the output states, no outcome analysis of any kind is done on the pilot extract.

5.8 Worked example: the pilot report (synthetic)

The candidate's pilot report to the supervisor, one page:

• Data. First extract failed two criteria (completeness 94.3%; two out-of-range values). Custodian confirmed: the duplicate was an export artefact; −15 should be 15 and 1,600 should be 160 (data-entry errors in the source system); ALDR is Alder Point's legacy code; 14 March was omitted from the export and was supplied; the walk-in count for 5 March and the waiting time for 12 March were never recorded. Corrected extract: completeness 98.6%; all criteria pass → go. Change to procedure: the codebook now records custodian-flagged downtime days and corrected values with the custodian's explanation.
• Interviews. Pilot interviews took 68 and 72 minutes against a 45-minute target. One question — "What mistakes were made during the diversion?" — made both pilot participants defensive. Changes: the guide was cut from 14 to 9 questions; the question was rewritten as "Walk me through the first twelve hours: what information did you have, and what did you decide at each point?" — anchored to time and information, which also reduces hindsight bias (Chapter 4). Re-piloted with one participant: 44 minutes → go.
• Pipeline. The analysis code ran end to end on synthetic data and produced all planned outputs except the event-study plot, which required a manual axis fix; the fix was moved into code → go.
• Deviation log entries: interview guide revised (before any study interview; no outcome data seen); downtime-day rule added (before any outcome data around the event were seen).

5.9 Common errors and limits

• Writing a proposal that cannot fail: no fixed elements, no stated criteria, no deviation rule.
• Ignoring elapsed-time dependencies; starting ethics and data agreements after the design is "perfect".
• A risk register without triggers or owners, never reviewed.
• Vague data requests ("all relevant data") that custodians cannot answer or that ask for more than the question needs.
• Using a pilot to estimate the effect or to "check whether there is anything there".
• Fixing data problems silently in a spreadsheet instead of asking the custodian and recording the fix.
• Deleting implausible days without understanding them — and thereby deleting the very disruption the study is about.
• Progression criteria written after the pilot results are seen.

The limit of a pilot is its size: passing it shows that procedures can work, not that the main data will be clean. Keep the audit script and run it on every extract in the main study.

Exercises — chapter 5

EX027 — Proposal summary. Objective: compress the study into a checkable summary. Inputs: Chapters 1–4. Constraints: 250 words; must name question, estimand, design, comparison, data sources, ethics status and expected contribution; no claims of results. Steps: (1) draft; (2) mark each sentence fixed or open; (3) ask a peer to state the study's primary estimate from your summary alone. Expected evidence: summary and the peer's restatement. Rubric: 3 = peer restates the estimand correctly, every required element present, nothing promised that the design cannot deliver; 2 = one element missing; 1 = topic-level; 0 = none.

EX028 — Timeline and risk register. Objective: plan against elapsed time. Inputs: §5.2–5.3. Constraints: a different study: interviews with twelve water-utility operators about cyber-incident exercises, plus analysis of three utilities' exercise reports. Steps: (1) build a four-semester timeline; (2) identify the critical path; (3) write six risks with likelihood, impact, mitigation, trigger and owner. Expected evidence: table and timeline. Rubric: 3 = critical path correctly identified (approvals and access to exercise reports), triggers measurable, mitigations specific to small utilities; 2 = one risk without trigger; 1 = generic; 0 = none.

EX029 — Pilot audit and re-pilot. Objective: apply a custodian's corrections to pilot extract data and decide GO or NO-GO on recorded completeness, without imputing values that were never recorded. Inputs: Lab 5 code. Constraints: offline. Steps: (1) run the lab and record the output; (2) apply the custodian's corrections from §5.8 (remove the duplicate; add 2025-03-14,AP,38,92,0,27; −15 → 15; 1,600 → 160; ALDR → AP; leave the two never-recorded values missing) and rerun; (3) report completeness and the decision; (4) state what the audit still reports and what you will do about it. Expected evidence: both outputs and a paragraph. Rubric: 3 = completeness 98.6% and GO reported, remaining missing cells and the 30 March low day explained with the downtime rule, no imputation in the pilot; 2 = one value wrong; 1 = not run; 0 = none.

EX030 — Interview guide pilot. Objective: test an instrument. Inputs: a draft guide of your own for RQ3 (at least ten questions). Constraints: pilot with one or two people outside the intended sample; record duration and every point of confusion or discomfort. Steps: (1) pilot; (2) revise; (3) document the changes and their reasons; (4) state whether the revision should be recorded in the deviation log. Expected evidence: original guide, revised guide and change log. Rubric: 3 = revisions respond to observed problems, questions anchored to time and information, deviation-log decision justified; 2 = revisions cosmetic; 1 = no pilot; 0 = none.

EX031 — Progression criteria. Objective: define go/amend/stop before seeing pilot results. Inputs: a feasibility pilot for a survey of 200 hospital IT managers about downtime procedures. Constraints: at least four criteria (for example response rate, item completion, time to complete, comprehension). Steps: (1) set thresholds for each zone; (2) justify each with the main study's needs; (3) state what "amend" changes for each. Expected evidence: table and justification. Rubric: 3 = thresholds tied to the main study's precision or feasibility, amend actions concrete, no effect-size criterion; 2 = one threshold arbitrary; 1 = criteria about results rather than feasibility; 0 = none.

EX032 — Supervisor memo on scope. Objective: make and record a scope decision. Inputs: after the pilot, the coalition offers patient-level stroke-care data for the diversion period if the candidate adds a stroke outcome. Constraints: one page; consider ethics, timeline, design and claim. Steps: (1) list benefits and costs; (2) decide; (3) write the deviation-log entry; (4) if declined, state how the offer becomes future work. Expected evidence: memo. Rubric: 3 = recognizes new ethics approval and individual-level data, timeline impact, multiplicity and the fixed primary outcome; decision defensible either way; deviation entry complete; 2 = decision without costs; 1 = accepts without approvals; 0 = none.

Chapter synthesis

A proposal is a contract: it fixes the question, estimands, design and analysis, declares what remains open, and states how deviations will be recorded. The critical path of a master's project runs through other people's time — ethics, custodians, legal review — so start those early and keep a living risk register. Ask custodians precise questions, especially about system changes, gaps and corrections. A pilot tests feasibility, not effects; never estimate an effect from it, and in an event study never look at outcomes around the event. Predeclare go/amend/stop criteria. Audit every extract with a script, turn each anomaly into a question, and treat downtime days at the very hospitals you study as measurement artefacts to flag, not rows to delete.

Continue studying — chapter 05 of 08

← Chapter 04 · Course chapter index · Chapter 06 →

Continue to the next authored lesson.