OpenCyber Academy
Postgraduate · Chapter 6 · English
Course code: OCA-152Chapter 06: OCA-152-C006

Chapter 6 — Application and software supply chain security: controlled evaluation

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 06 published. Map, example, EX026.

flowchart TD
  N0["Learning outcomes"]
  N1["6.1 Question"]
  N2["6.2 Design"]
  N3["6.3 Ethics"]
  N4["Lab — comparing detection times"]
  N5["6.4 Lab result"]
  N6["6.5 Interpreting"]
  N7["6.6 Other outcomes"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX026"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["6.1 Question"]
  N2["6.2 Design"]
  N3["6.3 Ethics"]
  N4["Lab — comparing detection times"]
  N5["6.4 Lab result"]
  N6["6.5 Interpreting"]
  N7["6.6 Other outcomes"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX026"]

[Learning outcomes]
    |
    v
[6.1 Question]
    |
    v
[6.2 Design]
    |
    v
[6.3 Ethics]
    |
    v
[Lab — comparing detection times]
    |
    v
[6.4 Lab result]
    |
    v
[6.5 Interpreting]
    |
    v
[6.6 Other outcomes]
    |
    v
[Worked example] --> [Exercise EX026]

Learning outcomes

You will (1) evaluate a new control with a controlled comparison; (2) choose outcome measures; (3) estimate uncertainty with the bootstrap; (4) interpret results cautiously.

6.1 Question

Does the automated SDK diff (Design B) shorten the time to detect unreviewed changes in third-party code?

The answer matters because the board's choice in chapter 1 assumed review would cut the chance of a repeat incident. If the diff does not detect changes faster than the old process, that assumption fails. Hannah writes the question down before the study starts, so the result cannot be reframed afterwards to fit whatever is found. Delivery delay and false alarms are recorded as secondary outcomes.

6.2 Design

Two mobile teams pilot Design B; two comparable teams continue with the old process for three months. Synthetic test changes, harmless but detectable (a new test-only network destination), are introduced into SDK updates in all four teams' test builds, with the teams' managers aware but not the engineers.

The planted changes are scheduled at random times across the three months, so neither group can predict them. Detection is timed from the moment a changed SDK reaches a team's test build to the moment someone on that team records it for review. Each team receives the same number of planted changes, so the groups face equal workloads.

6.3 Ethics

The engineers' managers and the works council approved the study. No real customer data is involved; results are reported by team, not by person.

Managers held the schedule of planted changes, so the engineers' detection reflected their normal routine rather than an alert they had been warned about. Data showing which engineer reviewed which change was kept by Hannah only and deleted at the end of the study. Teams were told in advance that a study was running, though not when changes would appear. Afterwards, every engineer in all four teams saw the results.

Lab — comparing detection times

"""Controlled evaluation: days to detect unreviewed SDK changes, pilot teams vs comparison teams, bootstrap (seed 3)."""
import random
from statistics import median
pilot = [1, 2, 1, 3, 2, 1, 4, 2]; comp = [9, 14, 6, 21, 11, 8, 30, 12]
rng = random.Random(3); d = sorted(median(rng.choices(comp, k=8)) - median(rng.choices(pilot, k=8)) for _ in range(4000))
print("median pilot", median(pilot), "comparison", median(comp), "difference", median(comp) - median(pilot))
print("95% interval", d[99], "to", d[3899])

Run with python3 lab_eval.py. Verified output:

median pilot 2.0 comparison 11.5 difference 9.5
95% interval 6.0 to 19.0

6.4 Lab result

Pilot teams detected test changes in a median of 2 days; comparison teams in 11.5 days, a difference of 9.5 days. The bootstrap 95% interval (4,000 resamples, seed 3) runs from 6.0 to 19.0 days, so the improvement is unlikely to be chance, though its size is uncertain.

The interval is wide because there were only eight planted changes per group, a point the next paragraph takes further. The lab script resamples the eight detection times in each group with replacement, recomputes the difference in medians each time, and reports the middle 95% of those differences. Hannah publishes the script alongside the result so audit can rerun it.

6.5 Interpreting

Eight changes per group is a small sample. Comparison teams knew a study was running, which might have changed their behaviour; if anything this would narrow the difference.

Planted changes may also be easier to spot than real ones, since each added a whole new network destination. Hannah reports these limits in the evaluation summary, and she recommends tracking real detections after rollout to confirm the effect outside the study. She also notes that the four teams were not chosen at random, so team differences could explain part of the gap between the groups.

6.6 Other outcomes

Delivery speed: pilot teams' median SDK update time rose by 1 day, as modelled. False alarms: 3 harmless changes flagged per month, each taking about 20 minutes to review.

The one-day rise matches the delivery model in chapter 5, which gives the mobile leads some confidence in the other estimates. Three false alarms a month means about an hour of review, which the teams judged acceptable. Hannah reports both figures beside the detection result so the CISO sees benefits and costs together. Neither figure gave the CISO a reason to delay the wider rollout.

Pre-registered measures

Before the pilot started, Hannah wrote down the outcome measures, the comparison and how she would decide success, and shared them with Dr Menon. This prevents choosing favourable measures after seeing results.

Worked example

The CISO approves rolling out Design B to all teams, with the same measures tracked quarterly.

Synthetic case. The study and data are synthetic.

Errors and limits

Planted test changes may be easier to spot than real ones; supplement with real-world monitoring.

Exercises

Synthesis

Evaluate controls with comparisons, uncertainty and side effects before scaling.

Continue studying — chapter 06 of 12

← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →

Worked answers

EX026. Does the automated SDK diff shorten time to detect unreviewed changes in third-party code?

EX027. Pilot 2 days, comparison 11.5, difference 9.5, interval 6.0 to 19.0.

EX028. Managers and the works council approved, no customer data was used and results were reported by team.

EX029. Update time rose by one day and there were about three false alarms a month taking 20 minutes each.

EX030. Only eight changes per group.