◇ OpenCyber Academy
Doctoral research · Chapter 4 · English

Chapter 4 — Baseline replication and pilot

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 4 — Baseline replication and pilot

Lesson map

Lesson 04 published. Map, example, EX021.

flowchart TD
  N0["Learning outcomes"]
  N1["4.1 Why a baseline"]
  N2["Lab — Excess privileges against role baselines in 60"]
  N3["4.2 Piloting excess privilege"]
  N4["4.3 Auditing the measure"]
  N5["4.4 Lessons from the pilot"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX021"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["4.1 Why a baseline"]
  N2["Lab — Excess privileges against role baselines in 60"]
  N3["4.2 Piloting excess privilege"]
  N4["4.3 Auditing the measure"]
  N5["4.4 Lessons from the pilot"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX021"]

[Learning outcomes]
    |
    v
[4.1 Why a baseline]
    |
    v
[Lab — Excess privileges against role baselines in 60]
    |
    v
[4.2 Piloting excess privilege]
    |
    v
[4.3 Auditing the measure]
    |
    v
[4.4 Lessons from the pilot]
    |
    v
[Worked example] --> [Exercise EX021]

Learning outcomes

You will (1) replicate a baseline measurement before the main study; (2) pilot an excess privilege measure; (3) audit measurement quality.

4.1 Why a baseline

Before testing effects, Yusuf must show that his measures work on Fernvale's data. He replicates the council's own reported figure for critical patch latency from the previous year using his extraction and finds his median within a day of the reported value. He investigates the small difference and finds the council excluded systems scheduled for retirement. He documents the exclusion and decides to follow it, so his baseline matches the definition that departments already understand. Fernvale's security lead confirms the exclusion rule in writing for the study.

Lab — Excess privileges against role baselines in 600 synthetic accounts

"""Measure excess privileges against role baselines in synthetic accounts (stdlib only)."""
import random
random.seed(17)
roles = {"clerk": {"read_cases", "edit_cases"}, "finance": {"read_ledger", "post_ledger"},
         "it_support": {"reset_password", "read_logs"}}
extra_pool = ["read_ledger", "post_ledger", "edit_cases", "admin_console", "read_logs", "export_all"]
accts = []
for i in range(600):
    r = random.choice(list(roles))
    perms = set(roles[r])
    for p in extra_pool:
        if p not in perms and random.random() < 0.06: perms.add(p)
    accts.append((r, perms))
excess = [(r, p - roles[r]) for r, p in accts if p - roles[r]]
n_extra = sum(len(e) for _, e in excess)
admin = sum(1 for _, e in excess if "admin_console" in e)
print(f"accounts 600; with excess privilege {len(excess)} ({100*len(excess)/600:.1f}%)")
print(f"excess grants {n_extra}; accounts holding admin_console without need {admin}")

Run with python3 lab_privilege.py. Verified output:

accounts 600; with excess privilege 152 (25.3%)
excess grants 166; accounts holding admin_console without need 34

4.2 Piloting excess privilege

The excess privilege measure is new to Fernvale, so Yusuf pilots it on six hundred synthetic accounts across three roles: clerk, finance and IT support. Each role has a baseline set of permissions. The lab finds 152 accounts, 25.3 percent, holding at least one permission beyond their baseline, with 166 excess grants in total. Thirty-four accounts hold the administrative console permission without needing it, which is the most serious finding because it allows configuration changes across the system. Yusuf reports these figures only to the security lead, not in public drafts.

4.3 Auditing the measure

Yusuf audits the pilot by taking a random sample of flagged accounts and asking department managers whether each excess permission is genuinely unneeded. Most are confirmed as unneeded. A few reflect temporary duties not recorded in the role baseline, such as a clerk covering finance during leave. He refines the measure to exclude permissions with an approved, unexpired exception, and records this refinement as a pre-specified definition change made before any outcome comparison, which the pre-registration permits for pilot-stage clarifications. The managers sign off.

4.4 Lessons from the pilot

The pilot shows that role baselines themselves are incomplete in some departments, which could make adopters look worse if adoption also improves baselines. Yusuf adds a check: baselines are frozen at the start of the study period, and later baseline changes are tracked separately. Prof. Hald notes that pilots often reveal that the measurement system is part of the intervention, and that separating the two is essential to interpreting any later difference between departments. Yusuf adds the frozen baselines to the data management plan and archives them with a checksum.

Worked example

Yusuf replicates the council's patch latency baseline within a day, pilots excess privilege finding 152 of 600 accounts (25.3 percent) with 166 grants and 34 unneeded admin console holders, audits flags and freezes role baselines.

Synthetic case. Accounts are synthetic; outputs come from lab_privilege.py.

Errors and limits

Letting baselines change during a study confounds the measure with the intervention.

Exercises

Continue studying — lesson 04 (chapter 04 of 08)

← Chapter 03 · Course chapter index · Chapter 05 →

Continue to the next authored lesson.