◇ OpenCyber Academy
Doctoral research · Chapter 3 · English

Chapter 3 — Original contribution proposal

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 3 — Original contribution proposal

Lesson map

Lesson 03 published. Map, example, EX015.

flowchart TD
  N0["Learning outcomes"]
  N1["3.1 The contribution"]
  N2["3.2 Defining risky"]
  N3["3.3 Labelling and agreement"]
  N4["Lab — labelling agreement"]
  N5["3.4 Lab result"]
  N6["3.5 Decision rules"]
  N7["Sampling hard cases"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX015"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["3.1 The contribution"]
  N2["3.2 Defining risky"]
  N3["3.3 Labelling and agreement"]
  N4["Lab — labelling agreement"]
  N5["3.4 Lab result"]
  N6["3.5 Decision rules"]
  N7["Sampling hard cases"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX015"]

[Learning outcomes]
    |
    v
[3.1 The contribution]
    |
    v
[3.2 Defining risky]
    |
    v
[3.3 Labelling and agreement]
    |
    v
[Lab — labelling agreement]
    |
    v
[3.4 Lab result]
    |
    v
[3.5 Decision rules]
    |
    v
[Sampling hard cases]
    |
    v
[Worked example] --> [Exercise EX015]

Learning outcomes

You will (1) state a testable contribution; (2) define measures and labels; (3) establish reliable labelling of risky configurations; (4) set decision rules before data collection.

3.1 The contribution

Amara proposes two contributions: a measured comparison of exposure windows under scanning and admission enforcement in one organisation, and an open, reusable method for measuring exposure from audit records and configuration history.

The second contribution may outlast the first. A measured result from one organisation will be debated; a method that reconstructs exposure from audit logs and configuration history can be reused by other organisations to test the same question, which is why the archive and the synthetic data generator are part of the contribution, not supplements.

3.2 Defining risky

A configuration is "risky" if it violates selected Baseline or Restricted checks of the Kubernetes Pod Security Standards: privileged containers, host path volumes, running without runAsNonRoot, or allowing privilege escalation. Using a public standard makes the definition checkable by others.

The four checks were chosen because each maps to a concrete harm in SP 800-190's terms: a privileged container or host path volume can reach the node; running as root and allowing privilege escalation widen what a compromised process can do. Limiting the definition to these keeps labels reliable, and the panel can see exactly what "risky" excludes, such as resource limits or image provenance.

3.3 Labelling and agreement

Automated checks can misfire, so two reviewers independently label a sample of findings as real (R) or not real (N). Agreement is measured with Cohen's kappa, interpreted with the bands of Landis and Koch (1977), where values above 0.80 are "almost perfect".

For Amara, labelling matters because the main outcome counts only real risky configurations. If the automated check flagged a workload that genuinely needed host access for a documented reason, and that finding were counted as exposure, the scanning arm would look worse than it is. The label guide therefore states when a documented exception counts as not real.

Lab — labelling agreement

"""Cohen's kappa for two raters labelling synthetic misconfiguration findings (stdlib only)."""
a = "RRRRRRRRRRRRNNNNNNNNRRRNNNRRRRNNNNRRRRRRNNNNNNRRRR"
b = "RRRRRRRRRRRNNNNNNNNNRRRNNRRRRRNNNNRRRRRNNNNNNNRRRR"
n = len(a)
po = sum(x == y for x, y in zip(a, b)) / n
pe = sum((a.count(c) / n) * (b.count(c) / n) for c in "RN")
k = (po - pe) / (1 - pe)
print(f"items {n}, observed agreement {po:.2f}, chance {pe:.2f}, kappa {k:.2f}")

Run with python3 lab_kappa.py. Verified output:

items 50, observed agreement 0.94, chance 0.51, kappa 0.88

3.4 Lab result

Across 50 synthetic findings the reviewers agree on 94%, against 51% expected by chance, giving kappa 0.88, almost perfect agreement. The three disagreements are discussed and the label guide is clarified for workloads that need host access for a documented reason.

Kappa matters here because most findings are real: with such an uneven mix, two reviewers labelling almost everything R would agree often by chance alone. The proposal panel asked exactly this question, and Amara answered with the 51% chance agreement, which raw agreement hides and kappa removes.

3.5 Decision rules

Written before collection: primary outcome median exposure hours; comparison by bootstrap interval; no-effect rule from chapter 1; cost measured as rejection rate with a ceiling of 5% of deployments after the first fortnight.

Writing them first stops the analysis from choosing its own success. If the main result had been a median difference of an hour and a half, the no-effect rule would require Amara to report no effect, however tempting it would be to call it a modest improvement. The cost ceiling works the same way for the rejection rate, and the panel's condition adds one more rule: exception workloads are reported separately.

Sampling hard cases

Half the labelled sample is drawn from workloads with documented host-access needs, the hardest cases.

Worked example

The proposal panel asks why kappa rather than raw agreement. Amara shows that with an uneven mix of labels, chance agreement is high, so raw agreement flatters the labels; kappa corrects for that. The panel then approves the proposal with the condition recorded in 3.5.

Synthetic case. Labels and panel are synthetic.

Errors and limits

High agreement on an easy sample can hide disagreement on hard cases; sample deliberately.

Exercises

Synthesis

Testable claims need public definitions, reliable labels and rules set in advance.

Continue studying — chapter 03 of 08

← Chapter 02 (draft) · Course chapter index · Chapter 04 (draft) →