◇ OpenCyber Academy
Doctoral research · Chapter 5 · English

Chapter 5 — Main study and independent validation

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 5 — Main study and independent validation

Lesson map

Lesson 05 published. Map, example, EX027.

flowchart TD
  N0["Learning outcomes"]
  N1["5.1 Running the study"]
  N2["Lab — Removal comparison"]
  N3["5.2 Reading the result"]
  N4["5.3 Independent validation"]
  N5["5.4 Complexity and excess"]
  N6["5.5 Effect in context"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX027"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["5.1 Running the study"]
  N2["Lab — Removal comparison"]
  N3["5.2 Reading the result"]
  N4["5.3 Independent validation"]
  N5["5.4 Complexity and excess"]
  N6["5.5 Effect in context"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX027"]

[Learning outcomes]
    |
    v
[5.1 Running the study]
    |
    v
[Lab — Removal comparison]
    |
    v
[5.2 Reading the result]
    |
    v
[5.3 Independent validation]
    |
    v
[5.4 Complexity and excess]
    |
    v
[5.5 Effect in context]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX027]

Learning outcomes

You will (1) run the randomised comparison; (2) estimate the difference in removal rates with its uncertainty; (3) validate removals independently.

5.1 Running the study

Twelve business units across the three partners take part, six in each condition, randomised within partners. Each runs one review cycle over six weeks. Rhea follows the written protocol and logs every deviation, such as a delayed sitting or a reviewer on leave, with date and reason. Partners export entitlements before and after the cycle under the data agreement, and Rhea codes them with the frozen codebook. She does not intervene in decisions; her role is to provide materials, observe and measure, which keeps the comparison between procedures rather than between levels of researcher attention.

Lab — Removal comparison

"""Compare excess-entitlement removal under rubber-stamp and structured reviews on synthetic units (stdlib only)."""
import statistics as st
stamp = [0.04, 0.12, 0.03, 0.09, 0.15, 0.06]
struct = [0.31, 0.18, 0.35, 0.12, 0.29, 0.24]
d = st.mean(struct) - st.mean(stamp); sp = ((st.variance(stamp) + st.variance(struct)) / 2) ** 0.5
print(f"rubber-stamp mean={st.mean(stamp):.3f} structured mean={st.mean(struct):.3f} diff={d:.3f} cohen_d={d/sp:.1f}")

Run with python3 lab_trial.py. Verified output:

rubber-stamp mean=0.082 structured mean=0.248 diff=0.167 cohen_d=2.4

5.2 Reading the result

In the synthetic data, units using the usual review remove on average 8.2 percent of their coded excess, while structured units remove 24.8 percent, a difference of 16.7 percentage points. The standardised effect is large, but with six units per condition the estimate is uncertain, and Rhea reports an interval rather than a single figure. She also looks at spread: one structured unit removed only twelve percent, overlapping the usual-review range. A large average effect with visible variation suggests the procedure helps generally but depends on local factors, which she examines in interviews.

5.3 Independent validation

Removing more is only good if removals are correct. Two coders who do not know each unit's condition review a random sample of removed entitlements against role descriptions and interviews with line managers, classifying each removal as correct, wrong or unclear. Wrong removals, which had to be reversed, are counted as harm. In the synthetic study, structured units have slightly more reversals in absolute numbers but a similar rate of reversal per removal. Reporting both benefit and harm, measured by blinded coders, prevents the thesis from presenting a procedure as better when it simply removes more indiscriminately.

5.4 Complexity and excess

For H1, Rhea computes the rank correlation between each unit's complexity index and its baseline excess rate. Correlation is not causation: complex role models might cause excess, or both might stem from rapid organisational change. Rhea therefore avoids causal language, reports the correlation with its interval, and checks whether the pattern holds within each partner as well as across all units, since a relationship driven entirely by differences between organisations would mean something quite different from one present inside each. The analysis appears in chapter 7 alongside the synthesis.

5.5 Effect in context

An effect size only means something in context. Rhea compares the extra removal achieved by the structured review with the extra reviewer time it required, roughly eight minutes more per hundred decisions in the synthetic data. She also compares it with what auditors usually find, so partners can judge whether the improvement is meaningful. A procedure that removes a third more excess for a modest time cost is attractive; one that needs twice the time may not be sustained. Context turns a statistical result into a decision partners can make.

Worked example

Structured units remove 24.8 percent of coded excess against 8.2 percent under usual review, with similar reversal rates confirmed by blinded coders; the effect is large but uncertain with twelve units.

Synthetic case. Units, rates and reversals are synthetic.

Errors and limits

Counting removals without checking their correctness rewards a procedure that removes access indiscriminately.

Exercises

Synthesis

A randomised comparison with blinded validation of removals shows both benefit and harm, and context decides whether the benefit is worth its cost.

Continue studying — chapter 05 of 08

← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →