Chapter 1 — Advanced research integrity and methods
Lesson map
Lesson 01 published. Map, example, EX001.
flowchart TD
N0["Learning outcomes"]
N1["1.1 The case"]
N2["1.2 Why method matters"]
N3["1.3 Mixed methods"]
N4["1.4 Reliability of coding"]
N5["Lab — coder agreement"]
N6["1.5 Lab result"]
N7["1.6 Integrity"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX001"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["1.1 The case"]
N2["1.2 Why method matters"]
N3["1.3 Mixed methods"]
N4["1.4 Reliability of coding"]
N5["Lab — coder agreement"]
N6["1.5 Lab result"]
N7["1.6 Integrity"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX001"]
[Learning outcomes]
|
v
[1.1 The case]
|
v
[1.2 Why method matters]
|
v
[1.3 Mixed methods]
|
v
[1.4 Reliability of coding]
|
v
[Lab — coder agreement]
|
v
[1.5 Lab result]
|
v
[1.6 Integrity]
|
v
[Worked example] --> [Exercise EX001]
Learning outcomes
You will (1) frame an application-security research question that can be answered with evidence; (2) choose methods that fit software-ecosystem data; (3) measure the reliability of human coding; (4) apply integrity rules to vulnerability research.
1.1 The case
Yusuf Demir is a doctoral candidate at the fictional Eastmoor University, supervised by Prof. Helen Okafor. His partner is the fictional Larchwood Foundation, which runs a synthetic open-source package registry called "larch". In January, a flaw was fixed in a popular logging package, quillog, in version 2.4.1. Months later, many projects that depend on quillog still pinned older versions. Yusuf's thesis asks whether automated dependency-update tools shorten the time projects keep known-vulnerable versions, and why some projects lag even with such tools.
1.2 Why method matters
Supply-chain research is full of tempting shortcuts: counting advisories, scraping manifests and calling every old version "vulnerable". Kula et al. (2018) showed that many projects do not update library dependencies and that developers are often unaware of the vulnerabilities in them; Decan, Mens and Constantinou (2018) measured how long vulnerable npm packages stayed reachable through dependency networks. Both papers define their measures carefully. Yusuf adopts the same discipline: every term in his question (fix release, adoption, exposure) gets an operational definition before data is touched.
1.3 Mixed methods
Numbers show how long exposure lasts; interviews with maintainers explain why. Yusuf plans a quantitative study of registry metadata and a qualitative study of 20 maintainers, following the interview-based approach of Wermke et al. (2022), who studied security and trust practices in open-source projects. The two halves answer different questions and need different checks: the numbers need matched samples and intervals, the interviews need a codebook and agreement between coders. Yusuf plans how they will be joined before collecting either, so that interview themes can be compared with exposure in the same projects rather than set side by side afterwards.
1.4 Reliability of coding
Deciding whether a project actually reaches the vulnerable code is a judgement. Two coders label each case and agreement is measured with Cohen's kappa, interpreted with the bands of Landis and Koch (1977). For quillog, the judgement is whether a dependent project calls the logging feature whose default the fix changed. Yusuf's codebook gives each label a definition and an example, and coders work without seeing each other's labels. The kappa target of 0.70 was set in the protocol before any coding began.
Lab — coder agreement
"""Cohen's kappa for two coders labelling 20 synthetic advisories as reachable (R) or not (N)."""
a = "RRNNRNRRNNRNNRRNNNRN"
b = "RRNNRNRNNNRNNRRNRNRN"
n = len(a); po = sum(x == y for x, y in zip(a, b)) / n
pe = sum((a.count(c) / n) * (b.count(c) / n) for c in "RN")
k = (po - pe) / (1 - pe)
print(f"observed agreement {po:.2f}, chance {pe:.3f}, kappa {k:.2f}")
Run with python3 lab_kappa.py. Verified output:
observed agreement 0.90, chance 0.505, kappa 0.80
1.5 Lab result
On 20 synthetic advisories, the coders agree on 18 (0.90). Chance agreement is 0.505, so kappa is 0.80, "substantial" on the Landis–Koch scale, close to "almost perfect". The two disagreements become worked examples for a sharper codebook. The figure passes the protocol's target of 0.70, but Yusuf reports the two disagreements as well as the score. Both involved projects that import quillog only through another package, which the codebook did not address. A new rule for indirect use resolves them before main-study coding starts. The rule carries a date, so main-study coders know which version applied.
1.6 Integrity
Yusuf works only with public metadata and synthetic replicas; he never tests whether a vulnerability can be triggered in anyone's live system. Interviews have ethics approval, and quotations are anonymised. Findings about unpatched projects are reported in aggregate, so no individual maintainer is exposed. For quillog this means Yusuf never runs a project's code to check whether the flaw is reachable; reachability is coded from public source and manifests. Maintainers who agree to be interviewed are told how quotations will be anonymised and that their projects will not be named in results.
Worked example
Yusuf's first protocol: research question, definitions (fix release = first version listed as fixed in the advisory; adoption = first manifest change that excludes all vulnerable versions; exposure = days between), coding procedure with kappa target ≥ 0.70, interview guide and ethics approval number. Prof. Okafor asks him to pre-register the analysis plan before collecting data.
Synthetic case. Eastmoor University, Larchwood Foundation, the larch registry, quillog and all data are synthetic.
Errors and limits
Kappa depends on how common each label is; report raw agreement and label counts alongside it.
Exercises
- EX001 — Thesis question. State Yusuf's research question. Rubric: 3 = automated update tools shorten exposure; why some lag; 2 = partial; 1 = vague; 0 = none.
- EX002 — Exposure definition. How is exposure defined? Rubric: 3 = days from fix release to adoption excluding vulnerable versions; 2 = partial; 1 = vague; 0 = none.
- EX003 — Prior work. What did Kula et al. (2018) find? Rubric: 3 = many projects do not update; often unaware of vulnerabilities; 2 = partial; 1 = vague; 0 = none.
- EX004 — Kappa result. Report the lab's agreement figures. Rubric: 3 = 0.90 observed, 0.505 chance, kappa 0.80 substantial; 2 = partial; 1 = one; 0 = none.
- EX005 — Why kappa. Why not report raw agreement alone? Rubric: 3 = corrects for chance agreement; 2 = partial; 1 = vague; 0 = none.
- EX006 — Integrity safeguard. Give two integrity safeguards in Yusuf's study. Rubric: 3 = no live testing; ethics approval; anonymised; aggregate reporting (two); 2 = one; 1 = vague; 0 = none.
- EX007 — Pre-registration. Why pre-register the analysis plan? Rubric: 3 = prevents choosing analyses after seeing data; 2 = partial; 1 = vague; 0 = none.
Synthesis
Define terms, measure coding reliability and protect people before collecting any supply-chain data.