Chapter 5 — Proposal and pilot
Lesson map
Lesson 05 published. Map, example, EX027.
flowchart TD
N0["Learning outcomes"]
N1["5.1 How many requests"]
N2["5.2 The pilot extraction"]
N3["5.3 Writing the proposal"]
N4["5.4 Decision value"]
N5["Lab — Requests needed to detect a change in on-time"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX027"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["5.1 How many requests"]
N2["5.2 The pilot extraction"]
N3["5.3 Writing the proposal"]
N4["5.4 Decision value"]
N5["Lab — Requests needed to detect a change in on-time"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX027"]
[Learning outcomes]
|
v
[5.1 How many requests]
|
v
[5.2 The pilot extraction]
|
v
[5.3 Writing the proposal]
|
v
[5.4 Decision value]
|
v
[Lab — Requests needed to detect a change in on-time]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX027]
Learning outcomes
You will (1) estimate the sample needed to detect a change in a proportion; (2) pilot data extraction; (3) write a concise proposal.
5.1 How many requests
Hana asks whether 240 requests per period are enough. Using the baseline on-time rate of 0.74 and the usual settings of five per cent significance and eighty per cent power, the lab estimates the requests needed per period. Detecting a rise to 0.85 needs 211 requests, a rise to 0.90 needs 90 and a rise to 0.95 needs 46. With 240 per period she can detect a rise to 0.85 or more. A smaller improvement could be real but missed, and she writes that sentence into the proposal so that a null result is not misread as proof that the portal failed.
5.2 The pilot extraction
Before the main study, Hana pilots the metadata extract on one month of tickets. She finds two problems. Some tickets record the date the case was closed rather than the date the answer was sent, and some postal requests were logged only when scanned, days after arrival. Both errors would distort the timing outcome. She agrees corrections with Brookfield: completion means the date the answer was sent, and receipt means the date stamped in the post room. The EDPB access guidelines support counting from receipt, which confirms that her corrected definition matches the legal clock.
5.3 Writing the proposal
The proposal fits on six pages. It states the question, the outcomes and their justification, the design and rivals, the sample size, the ethics arrangements and the analysis plan. Hana includes a table that pairs each rival with its check and a timeline that ends three months before submission. Dr. Pritchard asks her to cut background and add one paragraph on what Brookfield will do with each possible result, because a study that cannot change a decision is hard to justify to a busy organisation that is lending staff time to the research.
5.4 Decision value
Hana writes the decision paragraph. If the portal improves on-time completion without harming quality, Brookfield will retire the email route for access requests and promote the portal. If it improves timing but harms quality, Brookfield will add a completeness check before answers are sent. If it changes nothing, Brookfield will investigate the postal process, the slowest channel. Each result leads to a specific action. She notes that writing the decision paragraph also forced her to clarify what counts as harm to quality, which made the measurement plan sharper and easier for the data protection officer to approve.
Lab — Requests needed to detect a change in on-time completion
# Requests per arm needed to detect a rise in on-time completion (two-sided alpha 0.05, power 0.80).
import math
za, zb = 1.959964, 0.841621
def n(p1, p2):
pb = (p1 + p2) / 2
return math.ceil((za * math.sqrt(2 * pb * (1 - pb)) + zb * math.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2 / (p1 - p2) ** 2)
for p2 in [0.85, 0.90, 0.95]:
print(f"on-time 0.74 -> {p2:.2f}: {n(0.74, p2):4d} requests per period")
Run with python3 lab_power.py. Verified output:
on-time 0.74 -> 0.85: 211 requests per period
on-time 0.74 -> 0.90: 90 requests per period
on-time 0.74 -> 0.95: 46 requests per period
Worked example
Hana finds that 240 requests per period detect a rise from 0.74 to 0.85 (211 needed), fixes two timing errors found in the pilot and ties every possible result to a Brookfield action.
Synthetic case. Ticket data and pilot errors are synthetic; EDPB Guidelines 01/2022 are real.
Errors and limits
Power formulas assume independent requests; repeat requesters break this slightly. A detectable change is not the same as an important one.
Exercises
- EX027 — Needed sample. How many requests per period detect a rise from 0.74 to 0.90? Rubric: 3 = 90;; 2 = about ninety;; 1 = wrong;; 0 = none.
- EX028 — Null result meaning. Why must a null result not be read as failure? Rubric: 3 = the study may miss a real improvement smaller than detectable;; 2 = mentions power;; 1 = treats null as proof;; 0 = none.
- EX029 — Pilot errors. Name the two timing errors the pilot found. Rubric: 3 = closure date instead of answer date; postal requests logged at scanning;; 2 = names one;; 1 = vague;; 0 = none.
- EX030 — Legal clock. Why count from the post-room stamp? Rubric: 3 = the deadline runs from receipt, as the guidelines confirm;; 2 = mentions receipt;; 1 = uses scan date;; 0 = none.
- EX031 — Proposal decision paragraph. What does the decision paragraph add? Rubric: 3 = ties each result to a concrete action at Brookfield;; 2 = mentions use;; 1 = omits;; 0 = none.
- EX032 — Detectable versus important. Distinguish a detectable change from an important one. Rubric: 3 = detectable depends on sample size; important depends on harm to requesters;; 2 = defines one;; 1 = confuses them;; 0 = none.
Synthesis
Chapter 5 sizes and pilots the study. Chapter 6 collects the data and analyses it with a bootstrap interval.
Continue studying — chapter 05 of 08
← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →