◇ OpenCyber Academy
Bachelor's · Chapter 34 · English

Chapter 34 — Comparative project

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 34 — Comparative project

Lesson map

Lesson 34 published. Map, example, EX100.

flowchart TD
  N0["Learning outcomes"]
  N1["34.1 The problem"]
  N2["34.2 Criteria"]
  N3["34.3 Trial"]
  N4["34.4 Results"]
  N5["34.5 Satisfaction"]
  N6["34.6 Recommendation"]
  N7["Fair comparison"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX100"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["34.1 The problem"]
  N2["34.2 Criteria"]
  N3["34.3 Trial"]
  N4["34.4 Results"]
  N5["34.5 Satisfaction"]
  N6["34.6 Recommendation"]
  N7["Fair comparison"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX100"]

[Learning outcomes]
    |
    v
[34.1 The problem]
    |
    v
[34.2 Criteria]
    |
    v
[34.3 Trial]
    |
    v
[34.4 Results]
    |
    v
[34.5 Satisfaction]
    |
    v
[34.6 Recommendation]
    |
    v
[Fair comparison]
    |
    v
[Worked example] --> [Exercise EX100]

Learning outcomes

You will (1) compare two approaches to a supply-chain problem; (2) define criteria; (3) run a fair trial; (4) recommend with evidence.

34.1 The problem

CampusEats must choose between two ways of keeping dependencies current: a bot that opens small weekly pull requests, or a monthly manual "update day".

Both approaches keep dependencies within their allowed ranges; they differ in how often changes arrive and how much work each step needs. Lena frames the choice as a comparison with agreed criteria, so the decision rests on evidence rather than on which members argue loudest. Lena prepares a short description of each approach, including what members would do each week or month, so everyone understands what is being compared before the trial begins.

34.2 Criteria

Time to apply high-severity fixes, member time spent, number of broken builds and member satisfaction.

The criteria are agreed before the trial starts. Fix time measures security benefit, member time and broken builds measure cost, and satisfaction shows whether members will keep using the approach. Lena defines each measure precisely, for example counting member time from the update log rather than estimates, so both approaches are measured the same way. Satisfaction is gathered with a short anonymous survey at the end of each period, asking the same three questions both times.

34.3 Trial

Each approach runs for six weeks in turn, with records kept.

Running them one after the other means the same members and the same code are compared, though the advisories that appear in each period differ. Lena notes this as a limitation. Six weeks is long enough to include several advisories and at least one monthly update day. Records are kept in the tracker as the trial runs, not reconstructed later. The order of the two periods is chosen by coin toss, and both run during term.

34.4 Results

Bot: high-severity fixes applied in a median of 3 days; 2 hours of member time per week; 3 broken builds, each fixed quickly. Manual: median 16 days; 5 hours on update day each month; 1 large broken build that took a day to fix.

The bot took more member time over a month, but it was spread in small pieces and fixes arrived much sooner. The manual approach was cheaper in hours, but its single broken build was large and hard to diagnose, because many packages changed at once. Lena reports both sides so the society can weigh them.

34.5 Satisfaction

Members preferred the bot, saying small changes were easier to review.

Several members also said the manual day was hard to schedule around exams, and one update day was postponed. A few members found the weekly pull requests repetitive, which suggested grouping minor updates for development-only tools. Lena reports the comments with the numbers, since satisfaction predicts whether an approach will still be followed after the people who chose it leave. Most members answered. Their comments are kept with the survey results.

34.6 Recommendation

Adopt the bot, keep a monthly review of major versions.

The recommendation follows from the fix time result, which matters most for security, and from members' preference. Lena states the cost honestly: the bot takes more total time each month, but in smaller pieces. The society adopts it, groups development-only updates to reduce noise and plans to review the decision after a year, using the same measures as the trial. Lena shares the results with Grace. Grace's office later uses the trial as an example for other projects.

Fair comparison

To keep the trial fair, the same members ran both approaches and the same tests applied. Records were kept by a member not involved in choosing the approach. Lena also logged unusual events, such as a week when the CI service was down, so they could be excluded or explained. A fair trial is one where the only intended difference is the approach being compared, and where the people recording results do not have a stake in either outcome.

Decision record

The final recommendation is written as a decision record: context, options compared, evidence, choice and review date. It links to the trial data, so future members can revisit the choice if tools or circumstances change.

Worked example

Lena's report includes a limitation: the bot trial ran during a quieter term period. She recommends checking the measures again during the busy exam term. The check confirms the result.

Synthetic case. Trial data are synthetic.

Errors and limits

Sequential trials can be affected by timing; repeat when conditions change.

Exercises

Synthesis

Compare approaches on agreed criteria and test the result under different conditions.

Continue studying — chapter 34 of 40

← Chapter 33 (draft) · Course chapter index · Chapter 35 (draft) →