Chapter 5 — Proposal and pilot

Lesson map

Lesson 05 published. Map, example, EX027.

flowchart TD
  N0["Learning outcomes"]
  N1["5.1 Proposal"]
  N2["5.2 Pilot department"]
  N3["5.3 Pilot findings"]
  N4["5.4 Applying lessons"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX027"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["5.1 Proposal"]
  N2["5.2 Pilot department"]
  N3["5.3 Pilot findings"]
  N4["5.4 Applying lessons"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX027"]

[Learning outcomes]
    |
    v
[5.1 Proposal]
    |
    v
[5.2 Pilot department]
    |
    v
[5.3 Pilot findings]
    |
    v
[5.4 Applying lessons]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX027]

Learning outcomes

You will (1) write and agree a proposal; (2) run a pilot in one department; (3) record and apply lessons.

5.1 Proposal

Jonah's proposal sets out the question, the literature gap, the staggered design, measures, analysis plan, data agreement, ethics approval, timeline and risks. Risks include delays to Halyard's rollout, missing log weeks and low interview participation, each with a planned response. Dr Solberg and Halyard's IT manager approve it before data collection begins. The proposal becomes the reference point for the rest of the project: any later change is compared with it, recorded with a reason and reported in the dissertation, so readers can see how the study evolved and why.

5.2 Pilot department

Before the full rollout, Halyard enrols one department of about forty staff, the claims team. Jonah uses this pilot to test his data pipeline rather than to estimate effects. He checks that weekly aggregates arrive on time, that ticket categories map correctly to his measures and that sign-in logs separate passkey and password attempts. He also interviews two desk staff about how enrolment questions reach them. The pilot is small enough that problems affect few people, yet real enough to reveal issues that synthetic testing missed.

5.3 Pilot findings

Three issues appear. Sign-in logs counted a cancelled passkey prompt as a failure, inflating failures; the identity team adds a separate outcome code. Some reset tickets were logged as general access issues, so desk staff agree a clearer category. And several staff enrolled only on a laptop, then could not sign in on a shared terminal in the claims office. Halyard responds by offering a hardware security key for shared terminals. Jonah records each issue with evidence and the fix, and notes which fixes change his measures.

5.4 Applying lessons

Lessons from the pilot are applied before the main rollout. The new outcome code means sign-in success must be defined again, so Jonah updates the data dictionary and reruns the pilot numbers under both definitions to show the effect of the change. The ticket category change applies to all departments from the same date, so it does not favour one group. The security key option becomes part of standard enrolment. Each change is recorded against the proposal, ensuring that the main study begins with a cleaner pipeline and a documented history of why it differs.

Worked example

The pilot in the claims team reveals cancelled prompts counted as failures, inconsistent ticket categories and shared-terminal problems; fixes are applied before the main rollout and recorded against the proposal.

Synthetic case. The claims team, issues and fixes are synthetic.

Errors and limits

Treating pilot results as the main finding mistakes a pipeline test for an effect estimate.

Exercises

• EX027 — Proposal risks. Name two risks listed in the proposal.
  Show marking criteria

  Rubric: 3 = rollout delay, missing log weeks, low interview participation (two);; 2 = one;; 1 = vague;; 0 = none.
• EX028 — Pilot aim. What is the pilot mainly testing?
  Show marking criteria

  Rubric: 3 = the data pipeline, not the effect;; 2 = pipeline;; 1 = effect;; 0 = none.
• EX029 — Cancelled prompts. How were cancelled passkey prompts mishandled?
  Show marking criteria

  Rubric: 3 = counted as failures, inflating failure rate;; 2 = failures;; 1 = ignored;; 0 = none.
• EX030 — Shared terminal fix. What fixed the shared-terminal problem?
  Show marking criteria

  Rubric: 3 = hardware security keys offered;; 2 = security key;; 1 = passwords;; 0 = none.
• EX031 — Definition change. How does Jonah handle the changed success definition?
  Show marking criteria

  Rubric: 3 = update dictionary, rerun pilot under both definitions;; 2 = update;; 1 = ignore;; 0 = none.
• EX032 — Category fairness. Why apply the ticket category change to all departments at once?
  Show marking criteria

  Rubric: 3 = avoids favouring one group;; 2 = fairness;; 1 = speed;; 0 = none.

Synthesis

A clear proposal and a small pilot reveal pipeline and practical problems early, and recording fixes keeps the study honest.

Continue studying — chapter 05 of 08

← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →