Chapter 24 — Applied research methods
Lesson map
Lesson 24 published. Map, example, EX070.
flowchart TD
N0["Learning outcomes"]
N1["24.1 The question"]
N2["24.2 Evidence"]
N3["24.3 Analysis"]
N4["24.5 Reporting"]
N5["24.6 Ethics"]
N6["Collecting the data"]
N7["Presenting results"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX070"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["24.1 The question"]
N2["24.2 Evidence"]
N3["24.3 Analysis"]
N4["24.5 Reporting"]
N5["24.6 Ethics"]
N6["Collecting the data"]
N7["Presenting results"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX070"]
[Learning outcomes]
|
v
[24.1 The question]
|
v
[24.2 Evidence]
|
v
[24.3 Analysis]
|
v
[24.5 Reporting]
|
v
[24.6 Ethics]
|
v
[Collecting the data]
|
v
[Presenting results]
|
v
[Worked example] --> [Exercise EX070]
Learning outcomes
You will (1) ask a focused question; (2) collect evidence from Brightwave records; (3) analyse simply; (4) report honestly.
24.1 The question
Diego asks: "Do projects receiving weekly update pull requests fix high advisories faster than projects updated manually?" The question comes from Diego's own observation that some projects close high advisories much faster than others. It is narrow enough to answer from records Brightwave already has, and its answer would change a practical decision: whether to give every active project weekly update pull requests. Diego writes it down before looking at the data so the analysis cannot drift towards a convenient finding.
24.2 Evidence
Task records give advisory dates and fix dates for each project and update method over one year. The task tracker already holds these dates because every audit match becomes a task with an owner and due date. Using existing records means no new data collection and no extra work for developers. Diego keeps only high advisories, so the comparison is between similar kinds of work, and he checks twenty records by hand against the original advisories. The year of records covers projects with each update method, and each record gives its project's method at the time.
24.3 Analysis
He compares median fix times: weekly-update projects 3 days, manual projects 12 days. Medians suit this data better than averages, because a few long-running fixes in neglected projects would pull an average upward. With weekly-update projects at 3 days and manual projects at 12, both groups' medians sit inside the 7-day target for highs only for the weekly group. Diego reports the number of advisories in each group alongside the medians so readers can judge how solid each figure is. Only the weekly group's median sits inside the 7-day target for highs.
24.4 Limits
Projects were not randomly assigned; manual projects are older and less active, which could explain part of the difference. Wermke et al. (2022) interviewed open-source projects about supply chain practices and show how context shapes adoption. Diego states this limit in the report itself rather than in a footnote. Older, less active projects may fix slowly for reasons unrelated to how they receive updates, such as having no active owner or harder code to change. Diego's earlier observation that matches in projects with no recent work took nearly three weeks points the same way. The comparison therefore shows an association that supports the recommendation, not proof that weekly pull requests alone cause faster fixes.
24.5 Reporting
The report states the question, data, result, limits and a recommendation. The recommendation follows from the result but is worded no more strongly than the limits allow. Diego proposes weekly update pull requests for active projects and says the older projects should be reassessed after their rebuild. The data, script and cleaning decisions are attached, so Aiko or the tutor can rerun the analysis and check each step. The report fits on two pages, with the chart, the limits and the recommendation on the first, so the team can read it before the meeting.
24.6 Ethics
Records about developers' work are summarised by project, not used to judge individuals. Grouping by project keeps the research about the update method rather than about who was slow. A developer whose project sits in the manual group usually inherited that setup and is not at fault. Diego removes developer names from the exported records before analysis, and the report shows only project-level figures. Summaries by project also make the report safe to share with the tutor, since no individual developer can be identified from the figures shown.
Collecting the data
Diego exports a year of task records from the tracker, keeps only high advisories, and checks twenty records by hand against the original advisories to make sure dates are correct. He finds two records with a fix date earlier than the advisory date, caused by fixes made before the advisory was published, and handles them separately rather than letting them distort the median. The data and the script are saved with the report so that Aiko can rerun them. A short appendix lists every decision made while cleaning the data, so readers can judge whether they agree with it.
Presenting results
Diego presents the report in ten minutes at the team meeting, showing one chart, the limits and the recommendation, then takes questions about the cleaned records and the two unusual cases.
Worked example
The report recommends weekly update pull requests for all active projects, with older projects included after their rebuild. Aiko accepts it, and the change is made the following month.
Synthetic case. Data and results are synthetic.
Errors and limits
A difference in records is not proof of cause.
Exercises
- EX070 — Question. What was the question? Rubric: 3 = weekly PRs vs manual fix speed; 2 = partial; 1 = vague; 0 = none.
- EX071 — Result. What were the medians? Rubric: 3 = 3 vs 12 days; 2 = partial; 1 = vague; 0 = none.
- EX072 — Limit. Name a limit. Rubric: 3 = not random; older projects manual; 2 = partial; 1 = vague; 0 = none.
Synthesis
Ask focused questions and report limits honestly.
Continue studying — chapter 24 of 30
← Chapter 23 (draft) · Course chapter index · Chapter 25 (draft) →