◇ OpenCyber Academy
Master's · Chapter 5 · English

Chapter 5 — Proposal and pilot

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 5 — Proposal and pilot

Lesson map

Lesson 05 published. Map, example, EX027.

flowchart TD
  N0["Learning outcomes"]
  N1["5.1 The proposal"]
  N2["5.2 Piloting data collection"]
  N3["Lab — Timelock window"]
  N4["5.3 Window results"]
  N5["5.4 Adjusting the plan"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX027"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["5.1 The proposal"]
  N2["5.2 Piloting data collection"]
  N3["Lab — Timelock window"]
  N4["5.3 Window results"]
  N5["5.4 Adjusting the plan"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX027"]

[Learning outcomes]
    |
    v
[5.1 The proposal]
    |
    v
[5.2 Piloting data collection]
    |
    v
[Lab — Timelock window]
    |
    v
[5.3 Window results]
    |
    v
[5.4 Adjusting the plan]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX027]

Learning outcomes

You will (1) finalise the proposal; (2) pilot data collection; (3) examine the timelock window with pilot data.

5.1 The proposal

Tariq's proposal combines the main observational study with the secondary aim about timelock windows. It states the question, definitions, data sources, codebook, analysis plan, ethics approvals and a timeline. Dr Larsen reviews it with a second academic, who asks for clearer handling of protocols that changed control type during the window; Tariq adds a rule to code exposure time under each control. A reviewed proposal settles disagreements before data collection rather than after. The revised proposal is signed and filed.

5.2 Piloting data collection

Tariq pilots coding on six protocols not in the main sample. Two coders classify control type and incidents independently and agree on all control types but disagree on one incident, a mistaken parameter change that caused losses but was quickly reversed. They agree it counts, since users were harmed, and the codebook gains an example. Pilots expose ambiguities that would otherwise introduce inconsistent coding across forty protocols. The pilot also shows that coding one protocol takes about forty minutes, which lets Tariq plan the main collection schedule realistically.

Lab — Timelock window

"""Does a timelock window exceed monitoring detection-and-response time? (synthetic, read-only)"""
WINDOW_H = 48
# hours from a queued privileged change to an independent monitor raising and confirming an alert
DETECT_H = [3, 5, 6, 8, 9, 11, 14, 18, 22, 26, 31, 40, 52, 70]

def main():
    inside = [d for d in DETECT_H if d <= WINDOW_H]
    print(f"queued changes reviewed {len(DETECT_H)}, window {WINDOW_H} h")
    print(f"confirmed within window {len(inside)}/{len(DETECT_H)} ({100 * len(inside) / len(DETECT_H):.1f}%)")
    s = sorted(DETECT_H); mid = len(s) // 2
    print(f"median detection {(s[mid - 1] + s[mid]) / 2:.1f} h, max {max(s)} h")
    for w in (24, 72):
        print(f"if window were {w} h: {sum(d <= w for d in DETECT_H)}/{len(DETECT_H)}")

if __name__ == "__main__":
    main()

Run with python3 lab_window.py. Verified output:

queued changes reviewed 14, window 48 h
confirmed within window 12/14 (85.7%)
median detection 16.0 h, max 70 h
if window were 24 h: 9/14
if window were 72 h: 14/14

5.3 Window results

Using fourteen synthetic queued privileged changes from Lumen's records, the script compares hours from queueing to a confirmed monitor alert with a 48-hour timelock. Twelve of fourteen, 85.7 percent, are confirmed within the window; the median is 16 hours and the longest 70 hours. With a 24-hour window only nine would be confirmed in time; with 72 hours all fourteen. Tariq notes that detection is not the end: users still need time to act after an alert. He reports the data source and its limits.

5.4 Adjusting the plan

The pilot suggests that the secondary aim should measure time to user notification as well as confirmation, so Tariq adds that variable with Lumen's agreement. He also records the two slow cases' reasons, both changes queued on a weekend when staff coverage was low, as context. Each change is recorded as a pre-collection amendment. Pilots are valuable precisely because they change the plan while change is still cheap. He also asks Lumen how weekend coverage is arranged, so the context is accurate rather than assumed.

Worked example

Proposal review adds an exposure-time rule; the pilot clarifies incident coding; 12 of 14 queued changes are confirmed within 48 hours (median 16 h).

Synthetic case. Protocols, changes and timings are synthetic.

Errors and limits

Assuming a timelock is long enough without measuring detection and response times gives false comfort.

Exercises

Synthesis

A reviewed proposal and a pilot that tests both coding and the timelock question make the main study sound.

Continue studying — chapter 05 of 08

← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →