OpenCyber Academy
Bachelor's · Chapter 33 · English
Course code: OCA-106Chapter 33: OCA-106-C033

Chapter 33 — Research methods II

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 33 published. Map, example, EX097.

flowchart TD
  N0["Learning outcomes"]
  N1["33.1 Question"]
  N2["33.2 Design"]
  N3["33.3 Confounders"]
  N4["33.4 Coding agreement"]
  N5["33.5 Analysis"]
  N6["33.6 Reproducibility"]
  N7["33.7 Prior work"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX097"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["33.1 Question"]
  N2["33.2 Design"]
  N3["33.3 Confounders"]
  N4["33.4 Coding agreement"]
  N5["33.5 Analysis"]
  N6["33.6 Reproducibility"]
  N7["33.7 Prior work"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX097"]

[Learning outcomes]
    |
    v
[33.1 Question]
    |
    v
[33.2 Design]
    |
    v
[33.3 Confounders]
    |
    v
[33.4 Coding agreement]
    |
    v
[33.5 Analysis]
    |
    v
[33.6 Reproducibility]
    |
    v
[33.7 Prior work]
    |
    v
[Worked example] --> [Exercise EX097]

Learning outcomes

You will (1) design a comparative study; (2) choose measures and controls for confounders; (3) analyse with simple statistics and agreement measures; (4) report reproducibly.

33.1 Question

Lena's next question: do student projects that adopt dependency update bots fix high-severity advisories faster than those that do not?

The question comes from the society's own experience with the bot and the comparison in the next chapter, but it asks whether the result holds across other student projects. Lena writes down her expected answer and the evidence that would contradict it before collecting any data. She also checks that the data she needs exists in each project's repository history and advisory records, so that the study is feasible before she asks anyone to take part.

33.2 Design

Compare 12 student projects at Hollowbrook: 6 with bots, 6 without, matched by size. Measure days from advisory publication to fix merged.

Matching by size controls for one obvious difference, since larger projects have more dependencies and more advisories. Each project's owners give consent, and fix times come from repository history, which already exists, so the study does not change anyone's behaviour. Lena defines "fix merged" precisely, so every project is measured the same way. She writes the definitions in the study plan and asks Grace to review them before any data are collected.

33.3 Confounders

Projects with bots may have more experienced members. Lena records team experience and checks whether it differs between groups.

If the groups differ, a faster fix time could reflect experience rather than the bot. Lena also notes other possible factors, such as whether a project has an active maintainer during holidays, and records them for each project. With only twelve projects she cannot adjust for many factors statistically, so she reports them openly and discusses how each could affect the result.

33.4 Coding agreement

Two people independently judge whether each fix fully addresses the advisory. Agreement is measured with Cohen's kappa and interpreted with Landis and Koch (1977).

Before coding, the two people agree written criteria and practise on examples not in the study. Disagreements are discussed and the agreed result recorded, but the kappa reported is the one from before discussion, which shows how clear the criteria really were. Lena also reports the number of cases each pair disagreed on, so the kappa can be understood in context.

33.5 Analysis

Medians are compared because fix times are skewed; individual values are shown.

A few very slow fixes would distort a mean, so the median gives a fairer picture of a typical fix. Showing every value lets readers see the spread and spot unusual projects. With six projects per group, Lena avoids claiming a precise difference and describes the pattern, noting which values drive it and whether the conclusion would change if any one project were removed. All results go in an appendix.

33.6 Reproducibility

Scripts and a synthetic version of the data are shared.

Real repository data cannot always be shared, since it could identify projects or members, so the synthetic version keeps the structure and approximate values without real names. The scripts produce every figure in the report from the data, and Lena tests them on a clean machine before publishing, so another student could rerun the analysis and check it. A short guide explains how to run the scripts and what each output shows, so the work can be reused.

33.7 Prior work

Kula et al. (2018) found developers often do not update dependencies; Lena's study asks whether tooling helps in student projects.

Kula and colleagues studied large professional ecosystems, so their findings may not apply to small student teams with short membership. Lena's study cannot settle the wider question, but it adds evidence from a setting the earlier work did not cover. She states this limit in the report and suggests repeating the study at another university. Her review also lists more recent work.

Ethics and consent

Project leads consent to their repositories' metadata being analysed; results are reported without naming projects. Only public metadata and fix dates are used; no code from projects is copied into the study. The study plan is approved through the university's ethics process before data collection begins.

Reporting the study

The write-up follows a simple structure: question, method, data, results, limitations and recommendations. Lena includes a table of each project's fix times without names, so readers can see the spread, and states clearly that the study shows association, not proof of cause.

Worked example

Results: median fix time 6 days with bots, 24 days without; experience levels were similar. Kappa for fix judgement was 0.82. Lena reports the difference as promising for a small sample and recommends bots to all student projects.

Synthetic case. Data are synthetic.

Errors and limits

Twelve projects cannot establish cause; matched comparison reduces but does not remove bias.

Exercises

Synthesis

Comparative studies need matched groups, reliability checks and honest scope.

Continue studying — chapter 33 of 40

← Chapter 32 (draft) · Course chapter index · Chapter 34 (draft) →

Worked answers

EX097. Team experience.

EX098. 6 days with bots, 24 without; kappa 0.82.

EX099. Because fix times are skewed.