Chapter 4 — Baseline replication and pilot
Lesson map
Lesson 04 published. Map, example, EX021.
flowchart TD
N0["Learning outcomes"]
N1["4.1 Replicating the baseline"]
N2["4.2 Running a pilot"]
N3["4.3 Refining instruments"]
N4["4.4 What the pilot does not show"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX021"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["4.1 Replicating the baseline"]
N2["4.2 Running a pilot"]
N3["4.3 Refining instruments"]
N4["4.4 What the pilot does not show"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX021"]
[Learning outcomes]
|
v
[4.1 Replicating the baseline]
|
v
[4.2 Running a pilot]
|
v
[4.3 Refining instruments]
|
v
[4.4 What the pilot does not show]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX021]
Learning outcomes
You will (1) replicate a baseline measurement; (2) run a pilot of the intervention; (3) refine instruments before the main study.
4.1 Replicating the baseline
Before changing anything, Mireille replicates Vantor's existing measures for three months. She confirms that monthly simulation click rates vary between eight and nineteen percent, mostly tracking lure difficulty, which supports her decision to fix difficulty. She also measures the baseline for her primary outcome: about one in four real suspicious emails is reported within thirty minutes. Replication shows that her measurement process works in this setting and gives the reference point every later claim depends on. She also records how many simulations were sent each month, so that later comparisons use the same base.
4.2 Running a pilot
Two departments not in the main study pilot the reporting-focused programme for six weeks. Mireille watches for practical problems. The report button works in desktop email but is hidden in the mobile app, so staff who read email on phones cannot easily report. The thank-you message arrives after two hours, not minutes, because of a mail rule. Staff find the weekly digest useful but too long. Pilots exist to find such problems cheaply, before they damage the main study. Each problem is logged with the date it was found.
4.3 Refining instruments
Mireille fixes the problems. The vendor adds the report button to the mobile app; the mail rule is corrected; the digest is cut to three examples. She also tests the burden survey and finds one question confusing, so she rewrites it. She records every change and its reason in a log that will be published with the thesis. Prof. Aalto notes that transparent records of changes before the main study build trust in the final results. The revised survey is tested again with five volunteers before the main study, and all of them understand every question.
4.4 What the pilot does not show
Mireille resists reporting pilot effects as findings. Two departments over six weeks cannot show whether the programme works, and the pilot departments volunteered, so they may be unusually engaged. She reports the pilot only as a feasibility check: the programme can run, the measures work, and the practical problems are fixed. Treating a pilot as evidence of effect is a common mistake that leads to overconfident main studies and disappointing replications. She writes this caution into the pilot report itself, so that nobody at Vantor quotes the early figures as proof that the programme works, and she asks the steering group to wait for the main study.
Worked example
Mireille replicates baseline click and reporting measures, pilots the programme in two departments, fixes three problems and reports the pilot only as feasibility.
Synthetic case. Figures and departments are synthetic.
Errors and limits
Volunteer pilots are unrepresentative. Baseline periods may include unusual events.
Exercises
- EX021 — Baseline click range. What range did baseline click rates show? Rubric: 3 = eight to nineteen percent, tracking lure difficulty;; 2 = range;; 1 = vague;; 0 = none.
- EX022 — Baseline reporting. What was baseline reporting of real suspicious emails? Rubric: 3 = about one in four within thirty minutes;; 2 = 25%;; 1 = vague;; 0 = none.
- EX023 — Mobile problem. What problem did the pilot find with mobile email? Rubric: 3 = the report button was hidden in the mobile app;; 2 = mobile;; 1 = vague;; 0 = none.
- EX024 — Feedback delay. Why did thank-you messages arrive late? Rubric: 3 = a mail rule delayed them by two hours;; 2 = mail rule;; 1 = vague;; 0 = none.
- EX025 — Change log. Why log every change before the main study? Rubric: 3 = transparency builds trust in final results;; 2 = trust;; 1 = vague;; 0 = none.
- EX026 — Pilot limits. Why is the pilot not evidence of effect? Rubric: 3 = two volunteer departments over six weeks cannot show it;; 2 = feasibility;; 1 = effect;; 0 = none.
Synthesis
Replication fixes the baseline, and a pilot finds practical problems, but neither should be mistaken for evidence of effect.
Continue studying — chapter 04 of 08
← Chapter 03 (draft) · Course chapter index · Chapter 05 (draft) →