Chapter 1 — Advanced research integrity and methods
Lesson map
Lesson 01 published. Map, example, EX001.
flowchart TD
N0["Learning outcomes"]
N1["1.1 The research problem"]
N2["1.2 Why integrity first"]
N3["1.3 Coding failure tickets"]
N4["Lab — coder agreement"]
N5["1.4 Reading agreement"]
N6["1.5 Data ethics"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX001"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["1.1 The research problem"]
N2["1.2 Why integrity first"]
N3["1.3 Coding failure tickets"]
N4["Lab — coder agreement"]
N5["1.4 Reading agreement"]
N6["1.5 Data ethics"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX001"]
[Learning outcomes]
|
v
[1.1 The research problem]
|
v
[1.2 Why integrity first]
|
v
[1.3 Coding failure tickets]
|
v
[Lab — coder agreement]
|
v
[1.4 Reading agreement]
|
v
[1.5 Data ethics]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX001]
Learning outcomes
You will (1) frame a doctoral question in endpoint patch management; (2) design for integrity, from pre-registration to coder agreement; (3) measure agreement on labelled data before analysis.
1.1 The research problem
Yusuf Adeyemi is a doctoral candidate at the fictional Calder University, supervised by Prof. Ingrid Solberg. His partner organisation is the fictional Harrowgate Health Trust, which runs about 1,000 clinical and office endpoints. The trust patches everything at once each month, and two problems recur: some endpoints stay unpatched for weeks, and a bad update occasionally breaks clinical applications across every ward at the same time. NIST SP 800-40 Rev. 4 describes patching as preventive maintenance and recommends planning for both speed and the risk of disruption. Yusuf asks whether staged, ring-based deployment shortens the time to reach most endpoints while reducing update failures, compared with flat deployment.
1.2 Why integrity first
Endpoint data are messy: devices are offline, tickets are labelled inconsistently and outcomes can be chosen after seeing results. Yusuf pre-registers his hypotheses, primary outcomes (95th-percentile days to patch and failure rate) and analysis plan with his supervisor before collecting main-study data. The pre-registration names the two sites, the six-month window, the primary outcomes and the bootstrap interval for the failure-rate difference. Prof. Solberg holds a dated copy, so any later change in outcome or method can be compared with what was planned, and the thesis reports such changes with reasons.
1.3 Coding failure tickets
Failure reasons in help-desk tickets are free text. Yusuf writes a codebook with four labels: reboot problems, disk space, application breakage and network. Two coders label 80 synthetic tickets independently. The four labels came from reading a separate set of tickets during the pilot, so the codebook reflects how Harrowgate's help desk actually describes failures. Each label has a short definition and an example, and the coders label without seeing each other's choices. Agreement on these 80 tickets decides whether the codebook is good enough for the main study, against the 0.6 kappa threshold set beforehand.
Lab — coder agreement
"""Cohen's kappa for two coders labelling synthetic patch-failure tickets (stdlib only)."""
import collections
a = ["reboot"]*30 + ["disk"]*12 + ["app"]*18 + ["network"]*10 + ["reboot"]*3 + ["app"]*4 + ["disk"]*3
b = ["reboot"]*30 + ["disk"]*12 + ["app"]*18 + ["network"]*10 + ["app"]*3 + ["reboot"]*4 + ["network"]*3
n = len(a); po = sum(x == y for x, y in zip(a, b)) / n
ca, cb = collections.Counter(a), collections.Counter(b)
pe = sum(ca[k] * cb[k] for k in set(a) | set(b)) / n**2
print(f"tickets {n}, observed agreement {po:.3f}, chance {pe:.3f}, kappa {(po-pe)/(1-pe):.3f}")
Run with python3 lab_kappa.py. Verified output:
tickets 80, observed agreement 0.875, chance 0.296, kappa 0.822
1.4 Reading agreement
Observed agreement is 0.875, chance agreement 0.296 and Cohen's kappa 0.822. By the bands of Landis and Koch (1977), this is almost perfect agreement. Disagreements cluster between reboot and application labels, so the codebook gains a rule: if the application fails only until a reboot, label it reboot. The new rule resolves the commonest disagreement by naming a test a coder can apply from the ticket: did the application work again after a reboot? Yusuf records the rule in the codebook with its date, so the main-study coders use the revised version and the change is traceable.
1.5 Data ethics
Ticket text can name staff and occasionally patients. The trust's information governance lead approves a process where tickets are redacted before leaving the trust, and Yusuf works only with redacted, synthetic-structure extracts. Redaction happens inside the trust, by trust staff, before any extract reaches Calder, so no patient or staff name ever leaves Harrowgate. The approval names the fields kept, such as ticket date, device category and failure text with names removed, and the retention period at Calder. Calder's own ethics committee reviewed the same process, which the thesis cites with its reference.
Worked example
Prof. Solberg asks why kappa rather than simple agreement. Yusuf shows that with an uneven label distribution, two coders could agree often by chance; kappa corrects for that, and 0.822 is well above the 0.6 threshold they pre-registered.
Synthetic case. Yusuf, Prof. Solberg, Calder and Harrowgate are synthetic.
Errors and limits
High agreement on a pilot set does not guarantee agreement on later tickets; re-check a sample during the main study.
Exercises
- EX001 — Research question. State Yusuf's research question. Rubric: 3 = ring vs flat: time to patch and failures; 2 = partial; 1 = vague; 0 = none.
- EX002 — Primary outcomes. Which primary outcomes were pre-registered? Rubric: 3 = 95th pct days to patch; failure rate; 2 = one; 1 = vague; 0 = none.
- EX003 — Kappa values. Report the lab's agreement figures. Rubric: 3 = 0.875, 0.296, 0.822; 2 = two; 1 = one; 0 = none.
- EX004 — Interpretation band. How do Landis and Koch classify 0.822? Rubric: 3 = almost perfect; 2 = partial; 1 = vague; 0 = none.
- EX005 — Codebook rule. Which rule was added after disagreements? Rubric: 3 = app fails until reboot → reboot; 2 = partial; 1 = vague; 0 = none.
- EX006 — Redaction. How are staff and patient names protected? Rubric: 3 = redacted before leaving trust, approved process; 2 = partial; 1 = vague; 0 = none.
- EX007 — Patch framing. How does SP 800-40r4 frame patching? Rubric: 3 = preventive maintenance balancing speed and disruption; 2 = partial; 1 = vague; 0 = none.
Synthesis
Integrity comes from fixing questions, outcomes and coding rules before seeing results.