OpenCyber Academy
Postgraduate · Chapter 6 · English
Course code: OCA-143Chapter 06: OCA-143-C006

Chapter 6 — Security governance, risk and compliance: controlled evaluation

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 06 published. Map, example, EX026.

flowchart TD
  N0["Learning outcomes"]
  N1["6.1 Design"]
  N2["6.2 Speed"]
  N3["Lab — Permutation test on review duration"]
  N4["6.3 Depth"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX026"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["6.1 Design"]
  N2["6.2 Speed"]
  N3["Lab — Permutation test on review duration"]
  N4["6.3 Depth"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX026"]

[Learning outcomes]
    |
    v
[6.1 Design]
    |
    v
[6.2 Speed]
    |
    v
[Lab — Permutation test on review duration]
    |
    v
[6.3 Depth]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX026]

Learning outcomes

You will (1) design a controlled comparison of governance processes; (2) test a difference with a permutation test; (3) check depth as well as speed.

6.1 Design

Daniel evaluates tiering on twenty upcoming supplier reviews. With the committee's agreement, ten are randomly assigned to the old questionnaire and ten to the tiered process. Random assignment makes the groups comparable on average, avoiding the risk that easier suppliers end up in one group. The outcome is days from request to completed review. To check depth, an independent reviewer from internal audit scores each completed review for whether it identified the key risks of that supplier. Blinding the auditor to which process produced each review prevents expectations about tiering from colouring the depth scores.

6.2 Speed

The lab compares mean days. The old process averaged thirty-nine point three days and the tiered process twenty-six point five, a difference of twelve point eight days. A permutation test reshuffles the twenty results five thousand times to see how often chance alone produces a difference this large. The one-sided p-value is about two in ten thousand, so chance is an unlikely explanation. Daniel reports the difference with the caution that ten reviews per group is a small sample. With only twenty reviews, one unusually slow supplier could move a group mean by several days, so the depth check in the next section matters.

Lab — Permutation test on review duration

"""Permutation test: days to complete supplier reviews, old vs tiered process (synthetic)."""
import random
random.seed(5)
old = [34, 41, 29, 52, 38, 45, 31, 47, 40, 36]
new = [22, 30, 19, 35, 27, 24, 33, 21, 28, 26]
obs = sum(old)/10 - sum(new)/10
pool = old + new; hits = 0
for _ in range(5000):
    random.shuffle(pool)
    if sum(pool[:10])/10 - sum(pool[10:])/10 >= obs: hits += 1
print(f"mean old {sum(old)/10:.1f} days; tiered {sum(new)/10:.1f} days; difference {obs:.1f}")
print(f"one-sided permutation p = {hits/5000:.4f}")

Run with python3 lab_perm.py. Verified output:

mean old 39.3 days; tiered 26.5 days; difference 12.8
one-sided permutation p = 0.0002

6.3 Depth

Speed alone would be misleading if reviews became shallower. The internal auditor, not told which process was used, scores depth. Both groups identified the key risks in most reviews, and the tiered group performed slightly better on high-tier suppliers because reviewers read SOC 2 exceptions rather than skimming questionnaires. Daniel concludes that tiering was faster without loss of depth in this sample. He recommends rollout with continued monitoring of depth scores during the first year. Monitoring depth scores during rollout guards against reviewers drifting into shallow reviews once speed becomes the visible target.

Worked example

Daniel randomises twenty reviews, finds tiering 12.8 days faster with permutation p of 0.0002 and confirms depth with blinded scoring.

Synthetic case. Review data are synthetic.

Errors and limits

Small samples. Blinding may be imperfect if reviewers recognise formats.

Exercises

Synthesis

Controlled evaluation with blinded depth checks shows whether faster processes stay thorough.

Continue studying — chapter 06 of 12

← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →

Worked answers

EX026. To make the groups comparable on average, so easier suppliers do not cluster in one group.

EX027. 39.3 days for the old process and 26.5 days for tiering.

EX028. About 0.0002.

EX029. To avoid bias in scoring the depth of each review.

EX030. No; it was slightly better on high-tier suppliers in this sample.