Chapter 37 — Evidence review seminar
Lesson map
Lesson 37 published. Map, example, EX109.
flowchart TD
N0["Learning outcomes"]
N1["37.1 Reviewing evidence"]
N2["37.2 Labelling alerts"]
N3["Lab — Agreement on labels"]
N4["37.3 Interpreting kappa"]
N5["37.4 Strengthening the evidence"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX109"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["37.1 Reviewing evidence"]
N2["37.2 Labelling alerts"]
N3["Lab — Agreement on labels"]
N4["37.3 Interpreting kappa"]
N5["37.4 Strengthening the evidence"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX109"]
[Learning outcomes]
|
v
[37.1 Reviewing evidence]
|
v
[37.2 Labelling alerts]
|
v
[Lab — Agreement on labels]
|
v
[37.3 Interpreting kappa]
|
v
[37.4 Strengthening the evidence]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX109]
Learning outcomes
You will (1) review evidence critically; (2) measure agreement on alert labels; (3) interpret kappa.
37.1 Reviewing evidence
In a seminar, Priya presents her early evidence to classmates and Dr Kowal, who question it. Why are there two weeks of before data instead of four at one clinic? Because logging started late, she explains, and she reports it as a limitation. Are guest-to-internal attempts after separation zero? Allowed attempts are zero; denied attempts continue at a low rate, mainly from devices looking for printers. Critical review exposes weaknesses early, while they can still be fixed or explained. She records every question and her answer.
37.2 Labelling alerts
To measure how many guest alerts need attention, Priya and the IT lead each label twenty alerts independently as needing attention or not. Labels depend on judgement: a burst of denied connections from one phone might be harmless printer discovery or a sign of infection. Independent labelling, without discussing each case, is essential; otherwise the two reviewers influence each other and agreement looks higher than it really is. Both use the same short written definition of an alert needing attention, so differences reflect judgement rather than different rules, and the twenty alerts are drawn at random from the four-week period.
Lab — Agreement on labels
"""Cohen's kappa for two reviewers labelling synthetic alerts (stdlib only)."""
a = "TTFTFFTTFTFFTTTFFTFT"
b = "TTFTFTTTFFFFTTTFFTFF"
n = len(a); po = sum(x == y for x, y in zip(a, b)) / n
pe = sum((a.count(c) / n) * (b.count(c) / n) for c in "TF")
k = (po - pe) / (1 - pe)
print(f"n {n} observed agreement {po:.2f} chance {pe:.3f} kappa {k:.2f}")
Run with python3 lab_kappa.py. Verified output:
n 20 observed agreement 0.85 chance 0.500 kappa 0.70
37.3 Interpreting kappa
The reviewers agree on seventeen of twenty alerts, 85 percent. Because both label about half the alerts as needing attention, chance agreement alone would be 50 percent. Cohen's kappa adjusts for this and gives 0.70. On the bands proposed by Landis and Koch, this is substantial agreement, enough to report the measure with confidence. Priya also reviews the three disagreements with the IT lead and writes a clearer labelling guide, which should raise agreement in later weeks. She reports kappa together with the raw agreement and the sample size.
37.4 Strengthening the evidence
After the seminar, Priya adds a short note to each limitation, describing its effect. The shorter before period at one clinic makes that clinic's comparison less certain, so she reports it separately. The continuing denied attempts are explained with examples. She adds the kappa result to support the alert measure. Evidence that has survived questioning is stronger than evidence nobody has examined, and the seminar improved her capstone more than another week of data would have. The revised evidence appendix is reviewed by Dr Kowal.
Worked example
Agreement is 0.85 with chance 0.50, giving kappa 0.70, substantial on the Landis and Koch bands.
Synthetic case. Labels are synthetic.
Errors and limits
Reporting raw agreement without adjusting for chance overstates reliability.
Exercises
- EX109 — Kappa value. What kappa does the lab report and how is it described? Rubric: 3 = 0.70, substantial per Landis and Koch;; 2 = 0.70;; 1 = 0.85;; 0 = none.
- EX110 — Chance adjustment. Why is 85 percent agreement not enough on its own? Rubric: 3 = chance gives 50 percent here; kappa adjusts;; 2 = chance;; 1 = it is enough;; 0 = none.
- EX111 — Independence of labels. Why must reviewers label alerts independently? Rubric: 3 = discussion inflates agreement;; 2 = bias;; 1 = speed;; 0 = none.
Synthesis
Critical review and agreement statistics make judgement-based evidence credible.
Continue studying — chapter 37 of 40
← Chapter 36 (draft) · Course chapter index · Chapter 38 (draft) →