Chapter 24 — Applied research methods
Lesson map
Lesson 24 published. Map, example, EX070.
flowchart TD
N0["Learning outcomes and connection"]
N1["24.1 From question to study"]
N2["24.2 Comparisons and their traps"]
N3["24.3 Collecting data in an operational setting"]
N4["24.4 Analysis that fits the data"]
N5["24.5 Reporting"]
N6["24.7 Practical activity: design a study"]
N7["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX070"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["24.1 From question to study"]
N2["24.2 Comparisons and their traps"]
N3["24.3 Collecting data in an operational setting"]
N4["24.4 Analysis that fits the data"]
N5["24.5 Reporting"]
N6["24.7 Practical activity: design a study"]
N7["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX070"]
[Learning outcomes and connection]
|
v
[24.1 From question to study]
|
v
[24.2 Comparisons and their traps]
|
v
[24.3 Collecting data in an operational setting]
|
v
[24.4 Analysis that fits the data]
|
v
[24.5 Reporting]
|
v
[24.7 Practical activity: design a study]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX070]
Learning outcomes and connection
After this chapter you should be able to:
- turn an operational question into a small, answerable study with a stated method;
- choose a comparison — before/after, with/without, paired — and recognize what each can and cannot show;
- collect data ethically and safely in an operational setting;
- report results with their limits so that a manager can act on them without over-reading them.
Technologists are often asked questions that sound simple — "did the new alert rule reduce response time?", "is RLY-02 slower since the upgrade?", "do our station visits find configuration drift?" — and answered with impressions. This chapter teaches the small, honest study that sits between an impression and a research project. It reads published methods to use them, not to produce new ones.
24.1 From question to study
Write the question so that data can answer it: population (which stations, alerts, hosts), measure (with definition — Chapter 9), comparison (against what), period and decision (what will be done differently depending on the answer). "Did the new first-action text for Rule C reduce time from alert to control-room call?" is answerable; "is monitoring better?" is not.
The Rule C question names its population (absence alerts), measure (minutes to the call) and decision (adopt the text or not).
"Is monitoring better?" names none of them.
24.2 Comparisons and their traps
- Before/after on the same system is easy and weak: anything else that changed in the period (staff, season, workload) can explain the difference.
- With/without — some stations or shifts get the change, others do not — is stronger if the groups are similar, and strongest if who gets the change is decided by a fair method (alternation, random choice) rather than by who volunteered.
- Paired comparisons — the same people or hosts measured under both conditions — remove differences between units.
Watch for regression to the mean: changes introduced after an unusually bad month often look effective because the next month would have been better anyway. Watch for pooling groups with different baselines (Chapter 9). And write down the analysis before seeing the results — the idea behind preregistration in research (Nosek et al., 2018) applies at small scale too: it stops you choosing the measure that flatters the change.
24.3 Collecting data in an operational setting
Use data the service already records where possible: alert and acknowledgement times, ticket timestamps, relay counts. If you need new data — timing a manual procedure, asking operators — get the agreement of the operational owner, make it voluntary for people, do not disrupt operations, and do not record more personal information than the question needs. The Menlo Report's principles — respect for persons, beneficence, justice and respect for law and public interest — are a useful check for any study involving people or systems in ICT (Menlo Report, 2012).
24.4 Analysis that fits the data
For small operational studies, simple methods are best and most honest: medians and percentiles (Chapter 9), differences between groups with the number of observations stated, intervals for proportions (Wilson), and a check of whether the result survives obvious alternative explanations (remove one unusual week; compare shifts separately). If the conclusion changes when one data point is removed, say so.
The Rule C study did this check: removing the carrier-outage week moved the old-text median from 14 to 11 minutes, still slower than 6.
Comparing shifts separately would have been the next check.
24.5 Reporting
A one-page study note has: question; method (population, measure, comparison, period, what was decided in advance); results with counts; limits and alternative explanations; and recommendation with the confidence it deserves. Separate observed results from inferences, as everywhere in this book.
For the Rule C study, the observed results are 11 old-text alerts with a median of 14 minutes and 12 new-text alerts with a median of 6; the inference is that the first action helps. The limits line names the small numbers, the awareness effect and the carrier-outage week.
24.6 Worked example: did the new first-action text help? (synthetic)
FSTS rewrote Rule C's alert to include a first action ("call the Water control room, then check carrier status"). The question: does it shorten time from alert to control-room call? The team alternated the old and new alert text by week for eight weeks across the on-call rota, so each technologist received both. Results: 11 alerts under the old text, median 14 minutes to call; 12 under the new text, median 6 minutes. Every technologist who received both had a shorter median under the new text. Limits: small numbers; technologists knew about the study (they may have tried harder); one old-text week included a carrier outage with many alerts at once. Removing that week, the old-text median is 11 minutes — still slower than 6. Recommendation: adopt the new text for all absence rules; re-measure in three months from routine data.
24.7 Practical activity: design a study
Choose one question from: (a) has RLY-02's p95 latency increased since relayd 2.4.1? (b) do quarterly station visits find drift that automated checks miss? (c) does the access review find fewer issues each quarter? Write the study design: question, population, measure, comparison, period, decision, data source, ethical and safety notes, analysis planned in advance.
For question (a), the comparison is ready-made: Chapter 27's canary measured RLY-03's p95 latency at 41 ms on relayd 2.4.1 against 37 ms on 2.4.0, for one night-time hour. A study would extend that to several weeks of routine data, include the morning peak and the weekly report runs, and decide in advance what increase would trigger a rollback.
24.8 Common errors and limits
- Questions that data cannot answer.
- Before/after comparisons presented as proof.
- Choosing the measure after seeing the data.
- Studies that disrupt operations or collect unnecessary personal data.
- Results reported without counts or limits.
- Treating a small study as settled; it is evidence for a decision, to be re-checked.
The Rule C study avoided the second error by alternating text by week rather than comparing before with after.
Asking technologists for their opinion after the trial would have added nothing the times did not show, and would have collected personal views the study did not need.
Exercises — chapter 24
EX070 — Make it answerable. Objective: reframe vague questions. Inputs: (a) "Is our patching good?"; (b) "Are suppliers a risk?"; (c) "Does the SOC help?" Constraints: each rewritten question must name population, measure, comparison and period. Steps: rewrite all three. Expected evidence: three questions. Rubric: 3 = specific, measurable, with comparison and period and a decision attached; 2 = one weak; 1 = still vague; 0 = none.
EX071 — Spot the weaknesses. Objective: critique a study. Inputs: "After the March outage we added Rule C. In April there were 30 % fewer telemetry complaints than in March, proving Rule C works." Constraints: four weaknesses. Steps: name each and propose a better design. Expected evidence: list and design. Rubric: 3 = before/after only, regression to the mean after an unusual month, complaints as a poor measure of detection, no counts or interval, other changes in the period; better design measures detection time on alerts with a comparison; 2 = three; 1 = one; 0 = none.
EX072 — Write the study note. Objective: report a small study. Inputs: Chapter 24 ("Worked example: did the new first-action text help?"). Constraints: one page; headings from Chapter 24 ("Reporting"). Steps: write the note. Expected evidence: note. Rubric: 3 = all headings; counts and medians; limits including awareness, small n and the outage week, with the sensitivity check; recommendation proportionate with a re-measurement plan; 2 = limits thin; 1 = advocacy; 0 = none.
Chapter synthesis
Turn operational questions into small studies with a stated population, measure, comparison, period and decision. Prefer comparisons that rule out obvious alternatives — alternation or pairing over before/after — and decide the analysis in advance. Collect data safely and ethically with the operational owner's agreement, analyse simply, check sensitivity, and report counts and limits so that the result is used with the confidence it deserves.
Continue studying — chapter 24 of 30
← Chapter 23 · Course chapter index · Chapter 25 →
Continue to the next authored lesson.