OpenCyber Academy
Doctoral research · Chapter 5 · English
Course code: OCA-202Chapter 05: OCA-202-C005

Chapter 5 — Main study and independent validation

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 05 published. Map, example, EX029.

flowchart TD
  N0["Learning outcomes"]
  N1["5.1 Double-coding"]
  N2["Lab 3 — agreement"]
  N3["5.2 Reading kappa"]
  N4["5.3 Adjudication"]
  N5["5.4 Independent validation"]
  N6["5.5 Sharing results in standard form"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX029"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["5.1 Double-coding"]
  N2["Lab 3 — agreement"]
  N3["5.2 Reading kappa"]
  N4["5.3 Adjudication"]
  N5["5.4 Independent validation"]
  N6["5.5 Sharing results in standard form"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX029"]

[Learning outcomes]
    |
    v
[5.1 Double-coding]
    |
    v
[Lab 3 — agreement]
    |
    v
[5.2 Reading kappa]
    |
    v
[5.3 Adjudication]
    |
    v
[5.4 Independent validation]
    |
    v
[5.5 Sharing results in standard form]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX029]

Learning outcomes

You will (1) organise blind double-coding of a corpus; (2) compute inter-rater agreement with Cohen's kappa; (3) adjudicate disagreements without contaminating the data; (4) arrange validation by a team that did not build the pipeline.

5.1 Double-coding

Two trained coders label each sampled event independently with the Diamond core feature that a key judgment concerns — adversary, capability, infrastructure or victim — and with outcome codes. Coders do not see each other's work until both are finished.

In the lab, ten events are coded; on the full sample of 340 judgments, the same procedure reached kappa 0.74 after one codebook revision. Independence matters because coders who discuss early items converge on each other's habits, which inflates agreement without improving the codebook.

Lab 3 — agreement

"""Lab 3 (ch 05): Cohen's kappa for two coders labelling synthetic events with a Diamond core feature."""
from collections import Counter
a = ["infra", "cap", "infra", "victim", "adv", "infra", "cap", "victim", "infra", "cap"]
b = ["infra", "cap", "cap",   "victim", "adv", "infra", "cap", "infra",  "infra", "cap"]
n = len(a)
po = sum(x == y for x, y in zip(a, b)) / n
ca, cb = Counter(a), Counter(b)
pe = sum(ca[k] * cb[k] for k in set(a) | set(b)) / n ** 2
kappa = (po - pe) / (1 - pe)
print(f"observed agreement={po:.2f} expected={pe:.2f} kappa={kappa:.3f}")
print("disagreements:", [i for i, (x, y) in enumerate(zip(a, b)) if x != y])

Run with python3 lab_kappa.py. Verified output:

observed agreement=0.80 expected=0.31 kappa=0.710
disagreements: [2, 7]

5.2 Reading kappa

Raw agreement of 0.80 sounds strong, but chance alone would produce 0.31 given how often each label appears. Kappa removes that chance component and gives 0.710, conventionally described as substantial. Disagreements at items 2 and 7 both involve infrastructure versus another feature — a hint that the codebook's definition of infrastructure needs an example about shared hosting.

Tomas's revision added that example, and agreement on the full 340-judgment sample reached kappa 0.74. The chance-agreement figure matters for the thesis: if one label dominated, raw agreement could be high while kappa stayed low, and reporting only the raw figure would have overstated how well the codebook works.

5.3 Adjudication

A third person resolves disagreements using the codebook, and both original codes are preserved. Changing the codebook mid-study requires a version note and, where possible, recoding of earlier items with the new definition.

Tomas's one codebook revision, the shared-hosting example for infrastructure, is the case this rule covers. It carries a version note in the archive, and items coded under the first version were recoded so that kappa on the full sample is computed under a single definition. The adjudicator's decisions are logged with the codebook rule they relied on, so that a later reader can see whether one coder was overruled systematically.

5.4 Independent validation

A separate team re-runs the analysis from the archived data and code without assistance. Agreement on headline numbers is the validation. Disagreement is not failure; it is information that must be reported.

In Tomas's study the partner lab reproduced the Brier score to three decimals but found one mis-merged file. He fixed it and documented the correction in the thesis rather than in a private note. That error would never have shown up in his own reruns, because his pipeline merged the files the same wrong way every time.

5.5 Sharing results in standard form

Where findings include structured indicators or relationships, STIX 2.1 objects allow others to parse them; however, aggregated research results are usually better shared as tables and code than as indicators.

Tomas's results are a calibration table, a phrase-to-probability mapping and code; none of them is an indicator. Wrapping the calibration table in STIX objects would add parsing work for readers without adding meaning, so his archive publishes CSV with a schema and keeps STIX for any future work that produces threat objects.

Worked example

On the full sample of 340 judgments (synthetic), coders reach kappa 0.74 after one codebook revision. The validation team at a partner lab reproduces the Brier score to three decimals but finds one mis-merged file; Tomas fixes it and documents the correction.

Synthetic case. Sample size, kappa and the partner lab are invented.

Errors and limits

Kappa depends on label prevalence; report raw agreement too. Validation by friends who helped build the pipeline is not independent.

Exercises

Synthesis

Code blind and twice, correct for chance, adjudicate transparently, validate independently and report every correction.

Continue studying — chapter 05 of 08

← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →

Worked answers

EX029. observed agreement 0.80, expected 0.31, kappa 0.710.

EX030. Some agreement happens by chance given how common each label is; kappa removes that expected agreement so the figure reflects real coder consistency.

EX031. Both disagreements involve infrastructure, so its definition is ambiguous; add a clarifying rule and an example (e.g. shared hosting) to the codebook.

EX032. Both coders' original codes, the adjudicated code, the rationale and the codebook version used.

EX033. No. Having written the parser, they share its assumptions and can miss the same errors; validation needs a team that did not build it.

EX034. Describe the mis-merge, how it was corrected, and whether any reported number changed and by how much.

EX035. When the results are aggregate statistics and coding rules rather than indicators or relationships; tables, codebooks and code serve readers better.