Chapter 6 — Data collection and analysis
Lesson map
Lesson 06 published. Map, example, EX033.
flowchart TD
N0["Learning outcomes"]
N1["6.1 Collecting the data"]
N2["6.2 The main estimate"]
N3["Lab — Bootstrap interval for the difference in exces"]
N4["6.3 Rival checks"]
N5["6.4 Cost"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX033"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["6.1 Collecting the data"]
N2["6.2 The main estimate"]
N3["Lab — Bootstrap interval for the difference in exces"]
N4["6.3 Rival checks"]
N5["6.4 Cost"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX033"]
[Learning outcomes]
|
v
[6.1 Collecting the data]
|
v
[6.2 The main estimate]
|
v
[Lab — Bootstrap interval for the difference in exces]
|
v
[6.3 Rival checks]
|
v
[6.4 Cost]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX033]
Learning outcomes
You will (1) collect governance data under a pre-registered plan; (2) estimate a difference with a bootstrap interval; (3) run the planned rival-explanation checks.
6.1 Collecting the data
Over six weeks, an independent reviewer judges 300 sampled accounts in the quarterly group and 300 in the annual group, using the role catalogue with its new shared-resource rule. Lucas samples 60 review records per group for the evidence test. He logs every step: sample draws with their seeds, judgements with dates and any accounts that could not be judged. Only four accounts lacked enough information, and he records them as missing rather than guessing, as the plan requires. The reviewer works without knowing which group each account came from.
6.2 The main estimate
The reviewer finds 18 excess accounts of 300 in the quarterly group and 39 of 300 in the annual group: rates of 0.060 and 0.130. The difference is 0.070, about seven percentage points. A bootstrap with 2,000 resamples gives a 95% interval from 0.023 to 0.117. The interval excludes zero, so the data are consistent with a real difference in these units, though its size is uncertain: it could be as small as two points or as large as twelve. Lucas reports the interval alongside the point estimate in every table and figure, so that readers never see the difference without its uncertainty.
Lab — Bootstrap interval for the difference in excess-access rates
"""Bootstrap difference in excess-access rate, quarterly vs annual review units (synthetic)."""
import random
random.seed(7)
q = [1]*18 + [0]*282 # quarterly units: 300 accounts, 18 excess
a = [1]*39 + [0]*261 # annual units: 300 accounts, 39 excess
diff = sum(a)/len(a) - sum(q)/len(q)
boots = []
for _ in range(2000):
bq = [random.choice(q) for _ in q]; ba = [random.choice(a) for _ in a]
boots.append(sum(ba)/len(ba) - sum(bq)/len(bq))
boots.sort()
print(f"excess rate annual {sum(a)/len(a):.3f}; quarterly {sum(q)/len(q):.3f}; difference {diff:.3f}")
print(f"95% bootstrap interval {boots[49]:.3f} to {boots[1949]:.3f}")
Run with python3 lab_boot.py. Verified output:
excess rate annual 0.130; quarterly 0.060; difference 0.070
95% bootstrap interval 0.023 to 0.117
6.3 Rival checks
Lucas runs his planned checks. Turnover is similar in the two groups, and when he compares only the six units with mid-range turnover the difference shrinks but remains. System counts are similar. Quarterly units completed training on time slightly more often, so diligence remains a plausible partial explanation. Quarterly reviews removed an average of nine permissions each, and reviewers spent about forty seconds per account, so rubber-stamping looks unlikely. The evidence tests found no deviations in 60 quarterly records and two in 60 annual records.
6.4 Cost
Quarterly review costs more time. Reviewers in quarterly units logged about four times as many hours per year as those in annual units, though each review was shorter because fewer changes accumulated. Lucas presents both benefit and cost without combining them into a single figure, because the value of removing excess access depends on what that access could expose. He suggests that Tessaly might adopt quarterly reviews for high-risk systems and keep annual reviews elsewhere, a risk-based compromise consistent with ISO/IEC 27002:2022 guidance.
Worked example
Lucas finds excess access of 0.060 versus 0.130, a difference of 0.070 with interval 0.023 to 0.117, and checks rivals and cost.
Synthetic case. All counts and costs are synthetic.
Errors and limits
Diligence remains a partial rival. Cost-benefit depends on asset value.
Exercises
- EX033 — Missing judgements. How did Lucas handle accounts that could not be judged? Rubric: 3 = recorded them as missing;; 2 = missing;; 1 = guessed;; 0 = none.
- EX034 — Group rates. What excess-access rates did each group show? Rubric: 3 = quarterly 0.060, annual 0.130;; 2 = 18 and 39;; 1 = vague;; 0 = none.
- EX035 — Bootstrap interval. Give the 95% bootstrap interval for the difference. Rubric: 3 = 0.023 to 0.117;; 2 = excludes zero;; 1 = vague;; 0 = none.
- EX036 — Turnover check. What happened to the difference among mid-turnover units? Rubric: 3 = it shrank but remained;; 2 = remained;; 1 = vanished;; 0 = none.
- EX037 — Rubber-stamp evidence. Why does rubber-stamping look unlikely? Rubric: 3 = nine removals per review and forty seconds per account;; 2 = removals;; 1 = vague;; 0 = none.
- EX038 — Risk-based option. Which compromise does Lucas suggest? Rubric: 3 = quarterly for high-risk systems, annual elsewhere;; 2 = risk-based;; 1 = vague;; 0 = none.
Synthesis
A pre-registered analysis yields an uncertain but real difference, with rivals and cost reported alongside.
Continue studying — chapter 06 of 08
← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →