Chapter 1 — Advanced research integrity and methods
Lesson map
Lesson 01 published. Map, example, EX001.
flowchart TD
N0["Learning outcomes"]
N1["1.1 The case"]
N2["1.2 Coding findings"]
N3["Lab — Coder agreement"]
N4["1.3 Reading agreement"]
N5["1.4 Integrity and disclosure"]
N6["1.5 Validity from the start"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX001"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["1.1 The case"]
N2["1.2 Coding findings"]
N3["Lab — Coder agreement"]
N4["1.3 Reading agreement"]
N5["1.4 Integrity and disclosure"]
N6["1.5 Validity from the start"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX001"]
[Learning outcomes]
|
v
[1.1 The case]
|
v
[1.2 Coding findings]
|
v
[Lab — Coder agreement]
|
v
[1.3 Reading agreement]
|
v
[1.4 Integrity and disclosure]
|
v
[1.5 Validity from the start]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX001]
Learning outcomes
You will (1) frame a doctoral question about smart-contract review quality; (2) build a reliable coding scheme for review findings; (3) set ethical rules for handling vulnerability data.
1.1 The case
Lucía Ferreyra is a doctoral candidate at the fictional Kestrel Institute of Technology, supervised by Prof. Henrik Aalto. Her lead partner is the fictional Harbourline Credit Union Consortium, which runs a permissioned Ethereum-compatible ledger for settlement between member credit unions, and two fictional open-source teams that build token and escrow contracts. Every contract, finding and figure in this book is synthetic. Her question is whether reviewing contracts against a structured checklist derived from public standards leaves fewer high-severity issues undetected than the usual unstructured expert review.
1.2 Coding findings
A study of review quality needs a way to label findings that two people apply identically. Lucía codes each finding into four categories drawn from the SWC registry and the OWASP Smart Contract Top 10: access control, external calls including re-entrancy, arithmetic and rounding, and other. Each category has a written definition, two positive examples, two near misses and a decision rule for overlaps, such as coding a missing permission check on an upgrade function as access control even when it involves an external call.
Lab — Coder agreement
"""Cohen's kappa for two coders labelling synthetic review findings (read-only)."""
from collections import Counter
CATS = ["access", "external-call", "arithmetic", "other"]
# 40 synthetic findings: (coder A label, coder B label)
PAIRS = ([("access", "access")] * 11 + [("external-call", "external-call")] * 9 +
[("arithmetic", "arithmetic")] * 8 + [("other", "other")] * 7 +
[("access", "other")] * 2 + [("external-call", "other")] * 2 + [("arithmetic", "access")])
def kappa(pairs):
n = len(pairs)
po = sum(a == b for a, b in pairs) / n
ca, cb = Counter(a for a, _ in pairs), Counter(b for _, b in pairs)
pe = sum(ca[c] * cb[c] for c in CATS) / n ** 2
return po, pe, (po - pe) / (1 - pe)
def main():
po, pe, k = kappa(PAIRS)
print(f"items {len(PAIRS)}, agreed {sum(a == b for a, b in PAIRS)}")
print(f"observed {po:.3f}, chance {pe:.3f}, kappa {k:.3f}")
for (a, b), n in sorted(Counter(p for p in PAIRS if p[0] != p[1]).items()):
print(f"disagree {a} -> {b}: {n}")
if __name__ == "__main__":
main()
Run with python3 lab_kappa.py. Verified output:
items 40, agreed 35
observed 0.875, chance 0.253, kappa 0.833
disagree access -> other: 2
disagree arithmetic -> access: 1
disagree external-call -> other: 2
1.3 Reading agreement
Two trained coders independently label forty synthetic findings and agree on thirty-five, an observed agreement of 0.875. Expected chance agreement from the category frequencies is 0.253, giving Cohen's kappa of 0.833, which Landis and Koch (1977) would call almost perfect, though Lucía treats such labels as rough guidance. Four of the five disagreements send a finding to other: two access and two external-call findings. That pattern shows the other category is absorbing borderline cases, so she tightens its definition before real coding begins.
1.4 Integrity and disclosure
Vulnerability data is dangerous: a description of an unfixed weakness in a deployed contract could help someone steal funds. Lucía's data agreement says that partners share findings only after fixes are deployed or the contract is retired, that all contract names and addresses are replaced with pseudonyms, and that no publication will include code or steps that would let a reader reproduce an attack. Findings are stored on an encrypted institute server with access logged. The ethics board approves the plan, including a rule for reporting any unfixed issue found during the study.
1.5 Validity from the start
Lucía separates three validity questions. Construct validity asks whether her four categories capture what practitioners mean by review quality, so she checks the codebook with two partner auditors. Internal validity asks whether any difference between procedures comes from the procedure rather than from reviewer skill or contract difficulty, which motivates random assignment and blinded checking later. External validity asks whether findings from a permissioned ledger and two open-source teams generalise to public decentralised finance, which she will discuss honestly rather than assume.
Worked example
Lucía writes the research question, the four-category codebook and the disclosure agreement, then runs the forty-item agreement exercise. Kappa of 0.833 and a clear pattern of findings drifting into other lead to a tighter definition before real coding.
Synthetic case. Lucía Ferreyra, Henrik Aalto, Kestrel Institute of Technology, Harbourline Credit Union Consortium and all contracts, findings and figures are synthetic; the SWC registry, OWASP Smart Contract Top 10 and Landis and Koch (1977) are real sources.
Errors and limits
Kappa summarises agreement but hides where disagreement lies; inspect disagreements, and remember that high agreement does not prove the categories are the right ones.
Exercises
- EX001 — Research question. State Lucía's research question. Rubric: 3 = whether checklist review leaves fewer high-severity issues than unstructured review;; 2 = checklist vs usual;; 1 = tools;; 0 = none.
- EX002 — Category sources. Which public sources shape the four categories? Rubric: 3 = SWC registry and OWASP Smart Contract Top 10;; 2 = SWC;; 1 = CVE;; 0 = none.
- EX003 — Kappa value. What kappa did the coders reach? Rubric: 3 = 0.833 from 35 of 40 agreed;; 2 = 0.83;; 1 = 0.875;; 0 = none.
- EX004 — Disagreement pattern. What pattern did the disagreements show? Rubric: 3 = four of five went to other;; 2 = other absorbs;; 1 = vague;; 0 = none.
- EX005 — Disclosure rule. When may partners share findings? Rubric: 3 = after fixes deployed or contract retired;; 2 = after fix;; 1 = anytime;; 0 = none.
- EX006 — Publication limit. What may never appear in publications? Rubric: 3 = code or steps that reproduce an attack;; 2 = attack steps;; 1 = names;; 0 = none.
- EX007 — External validity. What external validity concern does Lucía note? Rubric: 3 = permissioned ledger may not generalise to public DeFi;; 2 = generalisation;; 1 = vague;; 0 = none.
Synthesis
A defensible study of review quality starts with a precise codebook, measured coder agreement and strict disclosure rules for vulnerability data.