◇ OpenCyber Academy
Doctoral research · Chapter 4 · English

Chapter 4 — Baseline replication and pilot

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 4 — Baseline replication and pilot

Lesson map

Lesson 04 published. Map, example, EX021.

flowchart TD
  N0["Learning outcomes"]
  N1["4.1 Why replicate"]
  N2["4.2 Pilot sample"]
  N3["Lab — exposure days"]
  N4["4.3 Lab result"]
  N5["4.4 Investigating outliers"]
  N6["4.5 Protocol revisions"]
  N7["4.6 Reporting the pilot"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX021"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["4.1 Why replicate"]
  N2["4.2 Pilot sample"]
  N3["Lab — exposure days"]
  N4["4.3 Lab result"]
  N5["4.4 Investigating outliers"]
  N6["4.5 Protocol revisions"]
  N7["4.6 Reporting the pilot"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX021"]

[Learning outcomes]
    |
    v
[4.1 Why replicate]
    |
    v
[4.2 Pilot sample]
    |
    v
[Lab — exposure days]
    |
    v
[4.3 Lab result]
    |
    v
[4.4 Investigating outliers]
    |
    v
[4.5 Protocol revisions]
    |
    v
[4.6 Reporting the pilot]
    |
    v
[Worked example] --> [Exercise EX021]

Learning outcomes

You will (1) replicate a published measurement before extending it; (2) run a small pilot to test definitions and tools; (3) compute exposure from synthetic metadata; (4) revise the protocol from pilot lessons.

4.1 Why replicate

Before claiming anything new, Yusuf re-runs the style of measurement used by Decan et al. (2018) on a synthetic replica of their setting, checking that his scripts reproduce the expected pattern: most vulnerable packages are fixed upstream quickly, but dependents lag. Reproducing the known pattern on a replica shows that his scripts measure exposure the way the field does, before they are applied to quillog. If the replica had not shown dependents lagging, the fault would be in his pipeline, not in the field's finding. The replication scripts are archived with their output, so examiners can confirm the pipeline found the known lag before it measured quillog.

4.2 Pilot sample

Twelve projects that depended on quillog, six with an update bot and six without. The pilot checks whether exposure can be measured consistently from registry metadata and manifests, how long collection takes per project, and whether the coding rules work. Twelve projects are enough to find problems with definitions, but not to estimate an effect with any precision. The pilot sample is excluded from the main study. Six per group also kept the pilot to a few weeks of collection, leaving time for the protocol changes that followed.

Lab — exposure days

"""Exposure days: time from a fixed release to adoption by dependent projects (synthetic)."""
from statistics import median
# (project, uses_update_bot, days from fixed release to adopting it)
data = [("p01", True, 3), ("p02", True, 5), ("p03", True, 2), ("p04", True, 40), ("p05", True, 6), ("p06", True, 4),
        ("p07", False, 31), ("p08", False, 12), ("p09", False, 88), ("p10", False, 45), ("p11", False, 20), ("p12", False, 160)]
for bot in (True, False):
    d = [x[2] for x in data if x[1] == bot]
    print(f"bot={bot!s:5} n={len(d)} median={median(d)} days  max={max(d)}  within 7 days={sum(v <= 7 for v in d)}/{len(d)}")

Run with python3 lab_exposure.py. Verified output:

bot=True  n=6 median=4.5 days  max=40  within 7 days=5/6
bot=False n=6 median=38.0 days  max=160  within 7 days=0/6

4.3 Lab result

Bot users: median 4.5 days, maximum 40, five of six adopted within a week. Non-users: median 38 days, maximum 160, none within a week. Project p04 is the bot user with 40 days. These are pilot figures and serve to test the pipeline, not to answer H1. The one long bot-user case is the most informative row, because it shows what kind of explanation H2 must account for. The non-users' long tail, with a maximum of 160 days, shows why medians and not means are reported.

4.4 Investigating outliers

p04's bot opened a pull request on day 1, but tests failed because quillog 2.4.1 changed a configuration default; the maintainer fixed tests on day 40. This is exactly the kind of lag H2 predicts. Yusuf records the case in his pilot memo with the pull-request date, the failing test and the date it was fixed. It leads to a protocol change: recording bot pull-request dates as well as merge dates, and coding test failures on bot pull requests, so the main study can see this pattern systematically.

4.5 Protocol revisions

The pilot shows that "adoption" must allow range specifiers: a manifest saying ">=2.4.0" already permits the fix if the lockfile is updated. Yusuf adds lockfiles to the data. Without lockfiles, projects whose manifests already allowed the fixed version would look exposed until someone edited the manifest. Adding lockfiles measures what the project actually resolves. The revision is listed in the pilot memo with the other two, and it was approved before the main study began. The original specifier is kept beside the resolved version for checking.

4.6 Reporting the pilot

The pilot is reported as a feasibility check, not evidence for H1; six projects per group cannot support strong claims. Yusuf writes the pilot chapter around what changed in the protocol and why, with the figures given as illustrations. The wide interval on the pilot difference shows why: any reader who took 33.5 days as an estimate of the effect would be misled. Its value is in the three protocol revisions, each of which changed what the main study collected and is listed in the pilot memo.

Tooling checks

Yusuf checks his parsing scripts against hand-coded results for the twelve pilot projects before trusting them at scale. Two scripts misread pre-release version labels such as 2.4.1-rc1 as later than 2.4.1; the version comparison is rewritten to follow the ecosystem's published ordering rules and retested on a list of tricky cases.

Worked example

Yusuf's pilot memo lists three revisions: include lockfiles, record bot pull-request dates as well as merge dates, and code test failures on bot pull requests. Prof. Okafor approves the main study.

Synthetic case. All pilot data are synthetic.

Errors and limits

Pilot effects are noisy; do not use them to set expectations for the main study's effect size.

Exercises

Synthesis

Replicate first, pilot small, and let outliers sharpen definitions.

Continue studying — chapter 04 of 08

← Chapter 03 (draft) · Course chapter index · Chapter 05 (draft) →