Chapter 33 — Research methods II
Lesson map
Lesson 33 published. Map, example, EX097.
flowchart TD
N0["Learning outcomes"]
N1["33.1 Analysing the mismatch data"]
N2["33.2 Reliability of coding"]
N3["33.3 Reporting with limitations"]
N4["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4
N4 --> EX["Worked example"]
EX --> Q["Exercise EX097"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["33.1 Analysing the mismatch data"]
N2["33.2 Reliability of coding"]
N3["33.3 Reporting with limitations"]
N4["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX097"]
[Learning outcomes]
|
v
[33.1 Analysing the mismatch data]
|
v
[33.2 Reliability of coding]
|
v
[33.3 Reporting with limitations]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX097]
Learning outcomes
You will (1) analyse data with appropriate statistics; (2) assess reliability of coding; (3) report results with limitations.
33.1 Analysing the mismatch data
Sofia returns to her research question with a year of synthetic reconciliation logs. She finds 14 mismatch events across 52 weeks. She classifies their causes: 6 late approvals, 4 extract timing gaps, 3 data-entry errors and 1 duplicated batch. She reports counts and proportions and plots mismatches over time, noting fewer after hourly reconciliation began. She avoids claiming the change caused the improvement, since other changes happened at the same time, such as new coordinator training. Careful analysis distinguishes what the data show from what one might hope they show.
33.2 Reliability of coding
Classifying causes involves judgement, so Sofia asks a classmate to classify the 14 events independently using the same definitions. They agree on 12. She calculates Cohen's kappa, a measure of agreement beyond chance, and interprets the result using Landis and Koch's 1977 guidelines, which describe ranges from slight to almost perfect agreement. They discuss the two disagreements and refine one definition. Checking reliability shows that her classification is not just personal opinion. Reporting the agreement figure lets readers judge how much weight to give her categories and conclusions.
33.3 Reporting with limitations
Sofia's report states its limitations: one year of data, a single cooperative, synthetic logs prepared for the course, and possible confounding from training changes. She explains how each limits the conclusions. She recommends the cooperative continue collecting data so later analysis can test whether hourly reconciliation truly reduces mismatches. Stating limitations openly strengthens a report, because readers can trust that the author has considered what could be wrong. Dr Asante says that a modest, well-supported conclusion is more valuable than a dramatic one that collapses under questioning by assessors.
Worked example
Sofia classifies 14 mismatches (6 late approvals, 4 timing gaps, 3 entry errors, 1 duplicate), agrees on 12 of 14 with a classmate and reports limitations.
Synthetic case. The logs and coding are synthetic; Landis and Koch (1977) is a real source.
Errors and limits
Claiming a change caused an improvement ignores other changes happening at the same time.
Exercises
- EX097 — Mismatch count. How many mismatch events were found in the year? Rubric: 3 = 14;; 2 = fourteen;; 1 = 52;; 0 = none.
- EX098 — Coding agreement. On how many events did the classifiers agree? Rubric: 3 = 12 of 14;; 2 = 12;; 1 = all;; 0 = none.
- EX099 — Confounder. What confounds the improvement after hourly reconciliation? Rubric: 3 = new coordinator training at the same time;; 2 = training;; 1 = vague;; 0 = none.
Synthesis
Careful analysis, reliability checks and stated limits make research conclusions trustworthy.
Continue studying — chapter 33 of 40
← Chapter 32 (draft) · Course chapter index · Chapter 34 (draft) →