Chapter 18 — Research methods I
Lesson map
Lesson 18 published. Map, example, EX052.
flowchart TD
N0["Learning outcomes"]
N1["18.1 Question from practice"]
N2["18.2 Choosing a method"]
N3["18.3 Ethics"]
N4["18.4 Data"]
N5["18.5 Analysis"]
N6["18.6 Results"]
N7["18.7 Interpretation"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX052"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["18.1 Question from practice"]
N2["18.2 Choosing a method"]
N3["18.3 Ethics"]
N4["18.4 Data"]
N5["18.5 Analysis"]
N6["18.6 Results"]
N7["18.7 Interpretation"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX052"]
[Learning outcomes]
|
v
[18.1 Question from practice]
|
v
[18.2 Choosing a method]
|
v
[18.3 Ethics]
|
v
[18.4 Data]
|
v
[18.5 Analysis]
|
v
[18.6 Results]
|
v
[18.7 Interpretation]
|
v
[Worked example] --> [Exercise EX052]
Learning outcomes
You will (1) form a research question from practice; (2) choose between survey, experiment and case study; (3) plan ethical data collection; (4) analyse simple data.
18.1 Question from practice
After the token leak, Lena asks: "Do pre-commit secret hooks reduce secrets pushed by student developers?"
The question comes from a real problem, can be answered within a year, and uses data the society already collects. Lena writes it down with what she expects to find and why, so that her results can later be compared with her expectations honestly rather than reshaped to fit them. Grace agrees that the question is narrow enough for a single student project and that the answer would change the society's practice either way.
18.2 Choosing a method
A survey could ask developers about their habits; an experiment could compare groups with and without the hook; a case study could examine CampusEats in depth. Lena chooses a small comparison across two terms plus a short survey.
The comparison gives numbers; the survey explains them. A full experiment with randomly assigned groups would be stronger but is not practical in a small society where everyone works in the same repository. Lena records why the other designs were not chosen.
18.3 Ethics
Participants consent; no individual results are reported; data are anonymised. The university ethics process approves the plan.
Developers can withdraw at any time without consequence, and survey answers are collected by someone outside the society so that nobody feels pressured. Lena stores the data on university systems and deletes it at the end of the study. Because the topic touches on mistakes people have made, she takes care that no report could identify who pushed a secret. The plan was approved in four weeks.
18.4 Data
Counts of secrets caught by the server-side scanner per term, before and after the hook, and survey answers about whether the hook was ever bypassed.
Counts come from the scanner logs, which existed before the study, so they are not influenced by it. Survey questions are tested on two volunteers first to check they are clear. Lena also records events that could affect the counts, such as the number of active developers each term, since more people writing code could raise detections regardless of the hook.
18.5 Analysis
Counts are small, so Lena reports them directly and avoids percentages that look precise.
With such small counts, a change from five to one could arise partly by chance. Lena states this plainly and compares the counts with the number of commits each term, which gives a rate that accounts for changes in activity. She avoids statistical tests that the data cannot support. Grace checks the analysis plan before Lena sees any data, so choices are not influenced by the results.
18.6 Results
Server-side detections fell from 5 to 1 per term; 3 of 14 developers said they had bypassed the hook at least once, usually when it flagged a harmless string.
The bypassed cases were not linked to the secrets that still reached the server, as far as the anonymised data could show. Lena reports the figures with the number of developers in each term and notes that the survey relied on memory. Lena also lists what the survey could not tell her, such as whether bypassing happened more often under deadline pressure.
18.7 Interpretation
The hook helps, but false positives cause bypasses, so the precision work from chapter 12 matters.
The study has limits: one society, two terms and small numbers. Lena recommends keeping the hook, improving its precision with an approved allow list and repeating the measurement next year. Her report suggests other student projects at Hollowbrook could try the same approach and compare results, which would give a stronger answer than a single small study can provide on its own. She shares the report with the society.
Literature first
Before collecting data, Lena reads what is known about developer behaviour and security tools. Johnson et al. (2013) found that developers avoid tools that produce many false positives and unclear messages. This prediction, that a noisy hook would be bypassed, turned out to be exactly what her survey found, and it guided the fix. Reading first saves effort and gives a study something to compare against.
Worked example
Lena presents the results at a society meeting. Members suggest an allowlist file for known harmless strings, with review. Bypasses are reported as zero the next term.
Synthetic case. Study data are synthetic.
Errors and limits
Two terms and 14 developers cannot support general claims; describe this as a local evaluation.
Exercises
- EX052 — Research question. State Lena's research question. Rubric: 3 = do pre-commit hooks reduce secrets pushed; 2 = partial; 1 = vague; 0 = none.
- EX053 — Results. Give the main results. Rubric: 3 = 5 to 1 detections; 3 of 14 bypassed; 2 = partial; 1 = vague; 0 = none.
- EX054 — Bypass fix. What reduced bypasses? Rubric: 3 = reviewed allowlist for harmless strings; 2 = partial; 1 = vague; 0 = none.
Synthesis
Small, ethical studies can test practical changes and reveal side effects.
Continue studying — chapter 18 of 40
← Chapter 17 (draft) · Course chapter index · Chapter 19 (draft) →