◇ OpenCyber Academy
Master's · Chapter 5 · English

Chapter 5 — Proposal and pilot

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 5 — Proposal and pilot

Lesson map

Lesson 05 published. Map, example, EX027.

flowchart TD
  N0["Learning outcomes"]
  N1["5.1 The proposal"]
  N2["5.2 Why sample review records"]
  N3["5.3 Piloting the sample"]
  N4["Lab — Exact upper bounds for attribute samples"]
  N5["5.4 Pilot lessons"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX027"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["5.1 The proposal"]
  N2["5.2 Why sample review records"]
  N3["5.3 Piloting the sample"]
  N4["Lab — Exact upper bounds for attribute samples"]
  N5["5.4 Pilot lessons"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX027"]

[Learning outcomes]
    |
    v
[5.1 The proposal]
    |
    v
[5.2 Why sample review records]
    |
    v
[5.3 Piloting the sample]
    |
    v
[Lab — Exact upper bounds for attribute samples]
    |
    v
[5.4 Pilot lessons]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX027]

Learning outcomes

You will (1) write a concise research proposal; (2) pilot evidence testing with attribute sampling; (3) interpret upper bounds on deviation rates.

5.1 The proposal

Lucas writes a twelve-page proposal: the question, a short review of standards and studies, the cross-sectional design, the outcome definition, the confounders, the rival explanations with planned checks, the ethics arrangements and a timeline of nine months. He includes the analysis plan as an appendix and a one-page summary for Tessaly's chief information security officer. The proposal states clearly that results describe Tessaly's units in one year and may not generalise to other companies, other systems or other kinds of review.

5.2 Why sample review records

Besides excess access, Lucas wants to know whether reviews actually happen as recorded. Checking every review record is too slow, so he borrows the auditor's tool of attribute sampling: draw a random sample of records, count deviations such as a missing sign-off, and compute an upper bound on the true deviation rate. If that bound is below a tolerable rate set in advance, the control is judged to operate effectively. He sets the tolerable rate at 5%, a common choice that he records with its rationale in the plan.

5.3 Piloting the sample

In the pilot, Lucas computes exact upper 95% bounds for several sample outcomes. With 60 records and no deviations, the bound is 0.049, just below the 5% tolerable rate. One deviation in 60 raises the bound to 0.077, and three deviations to 0.124. A sample of 25 with one deviation gives 0.176. Only the clean sample of 60 supports a conclusion of effective operation. The lab makes clear that small samples cannot support strong assurance, even when deviations are rare. He records each bound in his notes so that the choice of sample size can be checked later by his supervisor and by examiners.

Lab — Exact upper bounds for attribute samples

"""Attribute sampling: exact upper 95% bound on control deviation rate (stdlib only)."""
from math import comb
def cdf(k, n, p): return sum(comb(n, i)*p**i*(1-p)**(n-i) for i in range(k+1))
def upper(k, n):
    lo, hi = 0.0, 1.0
    for _ in range(60):
        mid = (lo+hi)/2
        if cdf(k, n, mid) > 0.05: lo = mid
        else: hi = mid
    return hi
for n, k in [(60, 0), (60, 1), (60, 3), (25, 1)]:
    print(f"sample {n}, deviations {k}: upper 95% bound {upper(k, n):.3f}")
print("tolerable rate 0.05 met:", [f"{n}/{k}" for n, k in [(60,0),(60,1),(60,3),(25,1)] if upper(k,n) <= 0.05])

Run with python3 lab_sample.py. Verified output:

sample 60, deviations 0: upper 95% bound 0.049
sample 60, deviations 1: upper 95% bound 0.077
sample 60, deviations 3: upper 95% bound 0.124
sample 25, deviations 1: upper 95% bound 0.176
tolerable rate 0.05 met: ['60/0']

5.4 Pilot lessons

The pilot changes the plan. Lucas decides to sample 60 review records per group rather than 25, because smaller samples cannot meet the tolerable rate even with a single deviation. He also clarifies what counts as a deviation: a review with no sign-off, a sign-off after the deadline, or a sign-off by someone not responsible for the unit. Dr. Solberg notes that this evidence test complements the excess-access measure: one shows whether reviews happened, the other whether access ended up correct. He updates the plan.

Worked example

Lucas pilots attribute sampling and finds only a clean sample of 60 gives a bound below 5%, so he sets n=60 per group.

Synthetic case. Sample outcomes are synthetic; the method is standard statistics.

Errors and limits

Sampling cannot detect rare systematic failures. Tolerable rates are judgement calls.

Exercises

Synthesis

Pilots set realistic sample sizes and sharpen definitions before the main study.

Continue studying — chapter 05 of 08

← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →