◇ OpenCyber Academy
Doctoral research · Chapter 4 · English

Chapter 4 — Baseline replication and pilot

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 4 — Baseline replication and pilot

Lesson map

Lesson 04 published. Map, example, EX021.

flowchart TD
  N0["Learning outcomes"]
  N1["4.1 Replicating the baseline"]
  N2["4.2 Pilot"]
  N3["Lab — pilot ring timings"]
  N4["4.3 Reading the pilot"]
  N5["4.4 Pipeline fixes"]
  N6["4.5 Configuration management"]
  N7["Clock and identity checks"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX021"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["4.1 Replicating the baseline"]
  N2["4.2 Pilot"]
  N3["Lab — pilot ring timings"]
  N4["4.3 Reading the pilot"]
  N5["4.4 Pipeline fixes"]
  N6["4.5 Configuration management"]
  N7["Clock and identity checks"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX021"]

[Learning outcomes]
    |
    v
[4.1 Replicating the baseline]
    |
    v
[4.2 Pilot]
    |
    v
[Lab — pilot ring timings]
    |
    v
[4.3 Reading the pilot]
    |
    v
[4.4 Pipeline fixes]
    |
    v
[4.5 Configuration management]
    |
    v
[Clock and identity checks]
    |
    v
[Worked example] --> [Exercise EX021]

Learning outcomes

You will (1) replicate the trust's baseline measures; (2) run a ring pilot; (3) check measurement and data pipelines.

4.1 Replicating the baseline

Before the pilot, Yusuf recomputes the trust's own patch-compliance report from raw deployment logs. His figures match the trust's to within one percentage point, confirming that he understands how the logs record installation and reboot. Agreement within one percentage point shows that Yusuf reads installation and reboot events the same way the trust does. Had it failed, any later difference between sites might reflect his reading of the logs rather than the deployment strategy. He keeps the recomputation script in the archive with its output.

4.2 Pilot

One month at the intervention site tests the ring mechanics and the data pipeline. The pilot is not part of the main analysis. It checks that rings advance as planned, that failure signals from ring 0 reach the desktop team in time, and that every endpoint's install and reboot events reach Yusuf's pipeline. Any change made during the pilot is recorded before the main study begins, so the six-month data run on fixed rules. The pilot ran at the intervention site only, so the comparison site's baseline was undisturbed.

Lab — pilot ring timings

"""Median days-to-patch per deployment ring in a synthetic pilot (stdlib only)."""
import statistics, random
random.seed(7)
rings = {"ring0 (IT, 40)": (40, 1, 3), "ring1 (pilot wards, 160)": (160, 3, 8), "ring2 (rest, 800)": (800, 7, 16)}
for name, (n, lo, hi) in rings.items():
    days = [random.randint(lo, hi) for _ in range(n)]
    print(f"{name}: median {statistics.median(days)} days, max {max(days)}")

Run with python3 lab_pilot.py. Verified output:

ring0 (IT, 40): median 2.0 days, max 3
ring1 (pilot wards, 160): median 6.0 days, max 8
ring2 (rest, 800): median 11.0 days, max 16

4.3 Reading the pilot

Ring 0 reaches median 2 days (max 3), ring 1 median 6 (max 8) and ring 2 median 11 (max 16). The waits between rings work as designed, and the pipeline captures each endpoint's install and reboot times. Ring 2's maximum of 16 days stays within the trust's 30-day policy, which answered the desktop lead's worry about waiting too long. These are pilot figures for one month and are used only to confirm the mechanics, not to estimate the effect. The pilot also showed the shut-down devices that lacked reboot times, which the next fix addressed.

4.4 Pipeline fixes

The pilot reveals that 3% of endpoints report no reboot time because they were shut down rather than restarted; Yusuf adds a rule treating the next boot as the reboot. Without the rule, shut-down devices would have looked unpatched until someone restarted them, inflating days to patch at both sites. The rule is applied equally to both sites and recorded in the analysis log. Yusuf also counts how many endpoints it affects in the main study, so readers can judge its influence.

4.5 Configuration management

The ring definitions are stored as configuration items following SP 800-128, so changes during the study are recorded. Storing rings as configuration items means that moving a device between rings, or changing a wait period, leaves a record with a date and approver. If the desktop team had changed ring membership midway, the analysis could identify which endpoints were affected. Yusuf checks the change record monthly during the study, and the thesis lists every change made with its reason. The record shows no ring change went unlogged.

Clock and identity checks

Deployment logs come from the management server and endpoint agents, whose clocks can differ. Yusuf compares server and endpoint timestamps on a sample and finds a median skew under one minute, small enough for day-level outcomes. He also checks that endpoint identifiers stay stable when devices are re-imaged; a re-imaged device keeps its asset tag, so he links records by asset tag rather than hostname.

These checks are recorded in a data quality note that later chapters cite.

Sign-off

The trust signs off the pilot report.

Worked example

The trust's desktop lead worries that ring 2 waits too long. Yusuf points out that ring 2's maximum of 16 days is still within the trust's 30-day policy, and the main study will test whether total time improves.

Synthetic case. Synthetic.

Errors and limits

A pilot shows feasibility, not effect; do not report pilot timings as results.

Exercises

Synthesis

Replicate the baseline and test the pipeline before measuring effects.

Continue studying — chapter 04 of 08

← Chapter 03 (draft) · Course chapter index · Chapter 05 (draft) →