Chapter 18 — Research methods I
Lesson map
Lesson 18 published. Map, example, EX052.
flowchart TD
N0["Learning outcomes"]
N1["18.1 Question"]
N2["18.2 Method"]
N3["18.3 Ethics"]
N4["18.5 Plan"]
N5["Designing the study"]
N6["Ethics check"]
N7["Sample size"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX052"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["18.1 Question"]
N2["18.2 Method"]
N3["18.3 Ethics"]
N4["18.5 Plan"]
N5["Designing the study"]
N6["Ethics check"]
N7["Sample size"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX052"]
[Learning outcomes]
|
v
[18.1 Question]
|
v
[18.2 Method]
|
v
[18.3 Ethics]
|
v
[18.5 Plan]
|
v
[Designing the study]
|
v
[Ethics check]
|
v
[Sample size]
|
v
[Worked example] --> [Exercise EX052]
Learning outcomes
You will (1) form a research question; (2) choose a method; (3) consider ethics.
18.1 Question
Do monthly patch reminders sent to staff reduce patch latency?
The question comes from the patch latency lab, where 26 of 45 laptops installed within 14 days and ten took over 30 days. Many delays came from staff postponing restarts, so a reminder seemed a cheap intervention. Aisha narrows the question to one change, a reminder email sent three days after each monthly release, so the answer can be measured. Gareth agrees to send the reminder from his own account, so staff recognise the sender and the study uses Bramley's normal communication channel.
18.2 Method
Compare latency for two months before and after reminders.
The comparison uses the management server's update reports for two monthly releases before the reminders and two after. The same 45 laptops are measured in each month, using the same definition of installed after restart. Aisha keeps each month separate in her table, so one unusual month can be seen instead of being hidden inside an average. Using existing reports means no new monitoring is installed on staff laptops for the study, which also keeps the ethics review light.
18.3 Ethics
Data are aggregated; no individual is named.
Aisha's tables show counts per month, such as 26 of 45, and never a named laptop owner. Raw reports with user names stay with Gareth inside IT, kept for the 90 days agreed for device reports. Staff are told that reminders are being tried and why, and Dr Grant confirms the light-touch ethics approval before data are pulled. Staff are also told that taking part needs no action beyond reading the reminders, and that no individual results will be reported to managers.
18.4 Limits
Other changes may affect results.
Aisha lists the likely confounders in advance: holidays, new laptops and any other policy change. The forced restart policy turns out to start in the same month as the reminders, which is exactly the kind of overlap she warned about. Writing the list early lets her name this limit plainly in chapter 33, not discover it afterwards. She also notes that four months is short, so a seasonal effect, such as holidays in one of the before or after months, could move the counts.
18.5 Plan
Pre-state the measure: share within 14 days.
The plan fixes three things before any data are seen: the measure, share of laptops installed within 14 days; the comparison, two months before and after; and the threshold, a rise of at least five laptops. Dr Grant signs the one-page plan, which stops Aisha from changing the measure later to suit the result. The plan also states what would count as no effect: a change of fewer than five laptops within 14 days, which she would report as inconclusive.
Designing the study
Aisha's study design is simple but explicit. The population is all 45 laptops. The outcome is the share patched within 14 days for each monthly update. The intervention is a reminder email sent three days after release. The comparison is two months before and two months after. Data come from the management server's update reports, which already exist, so no extra collection is needed.
She writes down in advance what result would count as improvement: a rise of at least five laptops within 14 days. Writing the threshold first prevents her from deciding afterwards that any change is meaningful.
She also lists possible confounders: holidays, new laptops and any other policy changes. Recording them early helps her interpret results honestly in chapter 33.
Ethics check
The study uses existing device data, aggregated, with no names. Dr Grant confirms it needs only light-touch ethics approval, which she obtains.
Sample size
Forty-five laptops is a small sample, so Aisha avoids percentages with false precision and reports counts beside them. She also plans to look at each month separately, since one unusual month could distort an average over only two months.
Project log entry
Aisha keeps a dated research diary for the reminder study. Each entry notes what data were pulled, any oddities, such as a laptop rebuilt mid-month, and decisions made. When she writes up chapter 33, the diary lets her explain every exclusion instead of reconstructing reasons from memory weeks later.
Review with the tutor
Dr Grant asks how she would know if the reminders annoyed staff enough to be ignored. Aisha adds a check on reminder open rates and a short end-of-study question, so a fading effect would show up rather than being hidden in averages.
Worked example
Aisha writes a one-page plan and Dr Grant approves it.
Synthetic case. Synthetic.
Errors and limits
Before-after designs cannot exclude other causes.
Exercises
- EX052 — Question. State the research question. Rubric: 3 = reminders reduce latency?; 2 = partial; 1 = vague; 0 = none.
- EX053 — Measure. Which measure was pre-stated? Rubric: 3 = share within 14 days; 2 = partial; 1 = vague; 0 = none.
- EX054 — Limit. What is the main limit? Rubric: 3 = other changes; 2 = partial; 1 = vague; 0 = none.
Synthesis
Plan simple studies carefully.
Continue studying — chapter 18 of 40
← Chapter 17 (draft) · Course chapter index · Chapter 19 (draft) →