◇ OpenCyber Academy
Master's · Chapter 5 · English

Chapter 5 — Proposal and pilot

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 5 — Proposal and pilot

Lesson map

Lesson 05 published. Map, example, EX027.

flowchart TD
  N0["Learning outcomes"]
  N1["5.1 The proposal"]
  N2["5.2 Linking certificates to incidents"]
  N3["5.3 Running the pilot"]
  N4["5.4 Fixing the instruments"]
  N5["5.5 What the pilot does not tell"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX027"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["5.1 The proposal"]
  N2["5.2 Linking certificates to incidents"]
  N3["5.3 Running the pilot"]
  N4["5.4 Fixing the instruments"]
  N5["5.5 What the pilot does not tell"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX027"]

[Learning outcomes]
    |
    v
[5.1 The proposal]
    |
    v
[5.2 Linking certificates to incidents]
    |
    v
[5.3 Running the pilot]
    |
    v
[5.4 Fixing the instruments]
    |
    v
[5.5 What the pilot does not tell]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX027]

Learning outcomes

You will (1) write a proposal that commits to outcomes, strata and analysis; (2) pilot data extraction on a small slice; (3) fix extraction errors before full collection.

5.1 The proposal

Felipe's proposal states the primary outcome, expiry incidents per 100 certificate-years, compared between manual and ACME renewal, overall and within owner-known and owner-missing strata. The secondary outcome is the share of ACME incidents preceded by an infrastructure change within seven days. The analysis plan uses rate ratios with bootstrap intervals, because incidents are few and their distribution is far from normal. He records the planned seed and resample count, and he lists in advance the inventory fields he will use, so that he cannot later search the dataset for a favourable split.

5.2 Linking certificates to incidents

The hardest technical step is linking tickets to certificates. Tickets mention hostnames in free text, sometimes misspelt, while the inventory uses fully qualified names. Felipe writes matching rules in advance: exact match on the subject alternative name first, then a documented list of known aliases, then manual review by two people for anything left. Each ticket must also mention certificate expiry or a browser or client error about validity; tickets that merely mention a certificate during an unrelated outage are excluded. The rules go into the proposal appendix.

5.3 Running the pilot

The pilot extracts three months of tickets and the matching inventory slice: 118 certificates and 11 candidate tickets. Two people apply the linking rules independently. They agree on nine tickets and disagree on two. One disagreement involves a ticket naming a load balancer that terminates TLS for several hosts; the other involves an expired intermediate certificate in a chain rather than the server certificate itself. Felipe also finds that 14 inventory rows lack a renewal-mode value, because older spreadsheets never recorded it. He treats these as instrument faults to fix, not as results to interpret.

5.4 Fixing the instruments

Felipe makes three rule changes. Load-balancer tickets are attributed to the certificate actually presented to clients, checked against the inventory's endpoint field. Expired intermediates are coded as expiry incidents attributed to the chain owner, because RFC 5280 path validation fails on any expired certificate in the path, not just the leaf. Rows without renewal mode are filled from the ACME client's own account records where possible, and otherwise marked unknown and reported as a separate group, not silently dropped. Each change is logged with a date before full extraction begins.

5.5 What the pilot does not tell

The pilot covers only three months, so Felipe does not compute or report incident rates from it. He uses it to estimate effort: linking took about four minutes per ticket, so a two-year extraction of roughly 120 tickets is feasible within a month. He also confirms that pseudonymisation runs at Briarwood before export and that the HMAC key never leaves their secret store. Dr. Solberg approves the revised proposal, and Felipe schedules full extraction for a quiet period so that infrastructure staff can answer questions about ambiguous tickets.

Worked example

Felipe commits to rates per 100 certificate-years with ownership strata, pilots linking on 118 certificates and 11 tickets, and fixes rules for load balancers, intermediates and missing renewal modes.

Synthetic note. The pilot counts and timings are invented; RFC 5280 path validation is real.

Errors and limits

A pilot that is too short cannot estimate rates and should not try. Changing rules after seeing outcome data, rather than during a pilot, becomes outcome-driven analysis.

Exercises

Synthesis

A specific proposal plus a pilot aimed at extraction faults makes the full dataset consistent before any rate is computed.

Continue studying — chapter 05 of 08

← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →