OpenCyber AcademyLesson map
Lesson 01 published. Map, example, EX001.
flowchart TD
N0["Learning outcomes"]
N1["1.1 The case"]
N2["1.2 What counts as misuse"]
N3["Lab — Coder agreement"]
N4["1.3 Reading agreement"]
N5["1.4 Integrity and disclosure"]
N6["1.5 Validity from the start"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX001"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["1.1 The case"]
N2["1.2 What counts as misuse"]
N3["Lab — Coder agreement"]
N4["1.3 Reading agreement"]
N5["1.4 Integrity and disclosure"]
N6["1.5 Validity from the start"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX001"]
[Learning outcomes]
|
v
[1.1 The case]
|
v
[1.2 What counts as misuse]
|
v
[Lab — Coder agreement]
|
v
[1.3 Reading agreement]
|
v
[1.4 Integrity and disclosure]
|
v
[1.5 Validity from the start]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX001]
Learning outcomes
You will (1) frame a doctoral question about cryptographic misuse in code review; (2) build and test a reliable coding scheme for misuse findings; (3) set integrity and disclosure rules for handling weaknesses in live payment systems.
1.1 The case
Nadia Haddad is a doctoral candidate at the fictional Larkspur University, supervised by Prof. Oskar Brandt. Her partner, the fictional Meridian Payments Cooperative, has 24 engineering teams that call vetted cryptographic libraries for card tokenisation, webhook signing and data-at-rest encryption. Every team, change and figure in this book is synthetic. Nadia's question is whether adding a cryptographic-misuse checklist and an automated lint step to code review reduces the number of confirmed misuse findings per ten merged changes, compared with Meridian's usual review. The unit of interest is misuse of correct libraries, not broken algorithms, which matches what practitioners most often get wrong.
1.2 What counts as misuse
Misuse means a correct primitive used in a way that voids its security guarantee. Katz and Lindell show that authenticated encryption is secure only under stated conditions, and SP 800-38D makes one explicit: a GCM nonce must never repeat under the same key. Nadia's codebook has four categories. Nonce-iv covers repeated, predictable or truncated nonces and initialisation vectors. Key-handling covers hard-coded, logged, over-shared or never-rotated keys. Primitive-choice covers outdated or unsuitable algorithms such as unsalted fast hashes for passwords. Other catches residual issues. Each category carries a definition, two examples, two near misses and an overlap rule.
Lab — Coder agreement
"""Cohen's kappa for two coders labelling synthetic cryptographic-misuse findings (read-only)."""
from collections import Counter
CATS = ["nonce-iv", "key-handling", "primitive-choice", "other"]
# 48 synthetic findings: (coder A label, coder B label)
PAIRS = ([("nonce-iv", "nonce-iv")] * 12 + [("key-handling", "key-handling")] * 13 +
[("primitive-choice", "primitive-choice")] * 10 + [("other", "other")] * 6 +
[("key-handling", "primitive-choice")] * 3 + [("nonce-iv", "primitive-choice")] * 2 +
[("other", "key-handling")] * 2)
def kappa(pairs):
n = len(pairs)
po = sum(a == b for a, b in pairs) / n
ca, cb = Counter(a for a, _ in pairs), Counter(b for _, b in pairs)
pe = sum(ca[c] * cb[c] for c in CATS) / n ** 2
return po, pe, (po - pe) / (1 - pe)
def main():
po, pe, k = kappa(PAIRS)
print(f"items {len(PAIRS)}, agreed {sum(a == b for a, b in PAIRS)}")
print(f"observed {po:.3f}, chance {pe:.3f}, kappa {k:.3f}")
for (a, b), n in sorted(Counter(p for p in PAIRS if p[0] != p[1]).items()):
print(f"disagree {a} -> {b}: {n}")
if __name__ == "__main__":
main()
Run with python3 lab_kappa.py. Verified output:
items 48, agreed 41
observed 0.854, chance 0.263, kappa 0.802
disagree key-handling -> primitive-choice: 3
disagree nonce-iv -> primitive-choice: 2
disagree other -> key-handling: 2
1.3 Reading agreement
Two trained coders independently label 48 synthetic findings and agree on 41, an observed agreement of 0.854. Chance agreement computed from each coder's category frequencies is 0.263, so Cohen's kappa is 0.802; Landis and Koch would call that substantial to almost perfect, but Nadia treats such labels as rough guidance. The disagreements are informative. Five of seven move a finding into primitive-choice, three from key-handling and two from nonce-iv. Coder B reads "a key derived with a fast hash" as an algorithm choice, while coder A reads it as key handling. Nadia adds an overlap rule: derivation of a key is key-handling unless the algorithm itself is deprecated.
1.4 Integrity and disclosure
Findings about live payment code are dangerous because a description of an unfixed weakness could help an attacker forge webhooks or decrypt stored tokens. Nadia's data agreement says Meridian shares a finding only after the fix has shipped or the code is retired, that repository names and key identifiers are replaced with pseudonyms, and that no key material, ciphertext sample or exploit procedure ever leaves Meridian. Findings sit on an encrypted university server with access logged. The ethics board also requires an escalation rule: any unfixed high-severity misuse Nadia notices during the study goes to Meridian's security lead within one working day, regardless of study arm.
1.5 Validity from the start
Nadia separates three validity questions before collecting real data. Construct validity asks whether "confirmed misuse per ten merged changes" captures what cryptographers mean by weaker security; she checks the codebook with two Meridian security engineers and against the failure modes listed in SP 800-57 Part 1 for key management. Internal validity asks whether any difference comes from the checklist rather than from team skill, which motivates random assignment of teams later. External validity asks whether results from one payments firm with mature libraries carry over to small start-ups, which she will argue cautiously rather than assume.
Worked example
Nadia writes the question, the four-category codebook and the disclosure agreement, then runs the 48-item agreement exercise. Kappa of 0.802 and a clear drift toward primitive-choice lead to one new overlap rule before real coding.
Synthetic case. Larkspur, Meridian and every finding are invented; SP 800-38D, SP 800-57 and Landis and Koch are real.
Errors and limits
Kappa summarises agreement but hides where disagreement lies; always tabulate the off-diagonal cells. High agreement also cannot prove that the categories are the right ones, only that two people apply them alike.
Exercises
- EX001 — Research question. State Nadia's research question in one sentence.
Show marking criteria
Rubric: 3 = whether checklist plus lint reduces confirmed misuse per ten merged changes versus usual review;; 2 = checklist versus usual without the measure;; 1 = vague "better crypto";; 0 = none. - EX002 — Defining misuse. How does the book define cryptographic misuse, and why is a GCM nonce rule an example?
Show marking criteria
Rubric: 3 = correct primitive used so its guarantee fails; nonce reuse under one key voids GCM security per SP 800-38D;; 2 = definition only;; 1 = "weak algorithm";; 0 = none. - EX003 — Agreement figures. Report observed agreement, chance agreement and kappa from the lab.
Show marking criteria
Rubric: 3 = 0.854, 0.263, 0.802;; 2 = kappa only;; 1 = wrong but plausible;; 0 = none. - EX004 — Drift pattern. Which category absorbed most disagreements, and what rule did Nadia add?
Show marking criteria
Rubric: 3 = primitive-choice (5 of 7); key derivation is key-handling unless algorithm deprecated;; 2 = category only;; 1 = rule only;; 0 = none. - EX005 — Sharing condition. Under what condition may Meridian share a finding with Nadia?
Show marking criteria
Rubric: 3 = only after the fix ships or the code is retired, pseudonymised;; 2 = after fix;; 1 = any time;; 0 = none. - EX006 — Escalation duty. What must Nadia do if she sees an unfixed high-severity misuse?
Show marking criteria
Rubric: 3 = tell Meridian's security lead within one working day, whatever the arm;; 2 = report it;; 1 = note it in data;; 0 = none. - EX007 — Internal validity threat. Name the internal-validity threat and the planned remedy.
Show marking criteria
Rubric: 3 = team skill confounds the effect; randomise teams;; 2 = threat only;; 1 = generic bias;; 0 = none.
Synthesis
A defensible study of crypto misuse starts with a precise codebook, measured coder agreement, explicit overlap rules and strict, ethics-approved handling of weaknesses in live systems.