OpenCyber AcademyLearning outcomes
You will (1) frame a falsifiable question about how organisations assess cyber risk; (2) set integrity rules for research inside a regulated organisation; (3) measure agreement between assessors who use a risk matrix.
1.1 A falsifiable question
Amara Lindgren is a doctoral candidate at the fictional Halvard University, supervised by Prof. Rosa Delgado. Her research partner is the fictional Corvane Regional Health Board, which runs four hospitals and keeps a risk register built on a five-by-five risk matrix. Amara asks whether replacing ordinal matrix ratings with calibrated quantitative estimates, in the style of FAIR described by Freund and Jones (2015), leads to more consistent and better-calibrated risk decisions. She writes down in advance which results would show her idea is wrong: no gain in agreement, calibration or decision quality.
1.2 Why matrices are contested
Amara reviews the case against risk matrices. Cox (2008) showed that matrices can assign identical ratings to risks that differ greatly in expected loss, and can even rank a smaller risk above a larger one, because ordinal bands discard information. Hubbard and Seiersen (2016) argued that cybersecurity risk can be measured with calibrated ranges and simple simulation. Standards are more neutral: ISO/IEC 27005:2022 and NIST SP 800-30r1 allow qualitative, semi-quantitative and quantitative approaches. Amara's contribution is therefore empirical: to test, inside one organisation, whether the quantitative approach performs better in practice.
1.3 Integrity rules
Research inside a health board carries risks: exposing real weaknesses, embarrassing named managers and influencing the decisions being studied. Amara's protocol, approved by the Halvard ethics board and Corvane's information governance committee, sets rules. All published scenarios are synthetic composites; no real vulnerability or patient data leave Corvane; assessors take part voluntarily and may withdraw; individual performance is never reported to managers; and her analysis plan is pre-registered before data collection. She also declares that she has no financial interest in any risk quantification product.
1.4 Agreement between assessors
Before comparing methods, Amara measures how consistently Corvane's assessors use the existing matrix. Two experienced assessors independently rate forty synthetic risks into Low, Medium and High bands. The lab computes observed agreement and Cohen's kappa, which corrects for agreement expected by chance. Kappa is 0.19, which Landis and Koch (1977) would describe as slight agreement. Amara treats this as a baseline problem rather than a criticism of the assessors: if two experts often disagree about the same risk, the register's priorities may depend more on who rated a risk than on the risk itself.
Lab — Agreement between two matrix assessors
# Agreement between two assessors rating 40 synthetic risks on a 5x5 matrix (High/Medium/Low bands).
import random
r = random.Random(11)
bands = ["Low", "Medium", "High"]
truth = [r.choice(bands) for _ in range(40)]
def rate(t):
if r.random() < 0.62: return t
return r.choice([b for b in bands if b != t])
a = [rate(t) for t in truth]; b = [rate(t) for t in truth]
po = sum(x == y for x, y in zip(a, b)) / 40
pe = sum((a.count(k) / 40) * (b.count(k) / 40) for k in bands)
print(f"observed agreement {po:.2f}; chance {pe:.2f}; kappa {(po - pe) / (1 - pe):.2f}")
Run with python3 lab_kappa.py. Verified output:
observed agreement 0.47; chance 0.35; kappa 0.19
Worked example
Amara frames a falsifiable comparison of matrices and quantitative estimates, sets integrity rules for research in a health board and measures a matrix-agreement baseline of kappa 0.19.
Synthetic case. Halvard, Corvane and all people are synthetic; Freund and Jones (2015), Cox (2008), Hubbard and Seiersen (2016), ISO/IEC 27005:2022, SP 800-30r1 and Landis and Koch (1977) are real.
Errors and limits
Pre-registration does not remove all bias. Kappa depends on how often each band occurs.
Exercises
- EX001 — Falsifiable plan. What makes Amara's question falsifiable?
Show marking criteria
Rubric: 3 = she states beforehand which results would show her idea is wrong;; 2 = prior statement;; 1 = vague;; 0 = none. - EX002 — Cox critique. What did Cox (2008) show about risk matrices?
Show marking criteria
Rubric: 3 = identical ratings for very different risks and possible rank reversals;; 2 = information loss;; 1 = vague;; 0 = none. - EX003 — Standards neutrality. What approaches do ISO/IEC 27005:2022 and SP 800-30r1 allow?
Show marking criteria
Rubric: 3 = qualitative, semi-quantitative and quantitative;; 2 = all three;; 1 = quantitative only;; 0 = none. - EX004 — Publication rule. What may appear in published scenarios?
Show marking criteria
Rubric: 3 = only synthetic composites;; 2 = synthetic;; 1 = real weaknesses;; 0 = none. - EX005 — Interest declaration. What does Amara declare?
Show marking criteria
Rubric: 3 = no financial interest in risk quantification products;; 2 = conflict;; 1 = vague;; 0 = none. - EX006 — Baseline kappa. What kappa did the two assessors reach?
Show marking criteria
Rubric: 3 = 0.19, slight agreement;; 2 = 0.19;; 1 = 0.62;; 0 = none. - EX007 — Baseline meaning. Why does low agreement matter for the register?
Show marking criteria
Rubric: 3 = priorities may depend on who rated the risk;; 2 = rater effect;; 1 = vague;; 0 = none.
Synthesis
Doctoral GRC research starts with a falsifiable question, strict integrity rules and a measured baseline.