Chapter 33 — Research methods II
Lesson map
Lesson 33 published. Map, example, EX097.
flowchart TD
N0["Learning outcomes"]
N1["33.1 Question"]
N2["33.2 Design"]
N3["33.3 Labels"]
N4["33.4 Results"]
N5["33.5 Validity"]
N6["33.6 Reporting"]
N7["Choosing pairs"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX097"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["33.1 Question"]
N2["33.2 Design"]
N3["33.3 Labels"]
N4["33.4 Results"]
N5["33.5 Validity"]
N6["33.6 Reporting"]
N7["Choosing pairs"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX097"]
[Learning outcomes]
|
v
[33.1 Question]
|
v
[33.2 Design]
|
v
[33.3 Labels]
|
v
[33.4 Results]
|
v
[33.5 Validity]
|
v
[33.6 Reporting]
|
v
[Choosing pairs]
|
v
[Worked example] --> [Exercise EX097]
Learning outcomes
You will (1) plan a more rigorous study; (2) compare groups fairly; (3) use reliable labels; (4) consider validity threats.
33.1 Question
Samuel asks whether teams that adopted the shared namespace template had fewer configuration findings per service than teams that did not, over one term.
The shared namespace template sets default-deny network policies, Restricted pod security and standard labels when a team creates a namespace. If it reduces findings, offering it to every team would be a cheap improvement. The question is narrow enough to answer with one term's data from Samuel's checker. Dr Jensen approved the plan before any data were collected, and the plan named the measure: confirmed findings per service over the term.
33.2 Design
Compare 12 adopting services with 12 non-adopting services of similar size and age, matching pairs to reduce confounding.
Matching pairs compares like with like, so a large old service is not compared with a small new one. Twelve pairs is small, but it covers most of StudyHub's services. Samuel writes the matching rules before looking at any findings, so the pairs cannot be chosen to produce a desired result. He also records the deployment counts for each pair, since busier services might have more findings.
33.3 Labels
Findings are labelled real or not by two reviewers independently; agreement is checked with Cohen's kappa, interpreted with Landis and Koch's (1977) bands.
Two independent reviewers reduce the chance that one person's judgement shapes the result. Kappa measures how much they agree beyond what chance would produce. Using the published bands gives a common language for describing agreement, though Samuel notes that the bands are conventions rather than strict thresholds. Disagreements were discussed after the independent labelling and resolved with a short written reason for each. The agreed labels, not either reviewer's original ones, were used in the analysis, and the reasons are kept with the data.
33.4 Results
Median findings per service: 1 for adopters, 4 for non-adopters. Reviewer kappa is 0.82, almost perfect agreement.
The difference in medians is large for a small study, and the high kappa means the labels behind it are consistent. Samuel presents the per-pair results in a table, so readers can see that the adopting service had fewer findings in most pairs, not just on average. He reports the pairs where the non-adopting service had fewer findings too, with possible explanations, rather than leaving them out. One such pair involved a very small service with few configuration options to get wrong, which suggests that service size still matters despite the matching.
33.5 Validity
Teams chose whether to adopt; more careful teams may adopt first. Matching helps but cannot remove this. One term is short.
Self-selection is the main threat: teams that adopted the template early may also be the ones that follow other good practices. Matching on size, age and team size cannot fix that. Offering the template to everyone and re-measuring, as the recommendation suggests, would give a stronger test. Samuel lists these threats in the report beside the result. A longer study would help too.
33.6 Reporting
Results with limits, plus a recommendation to offer the template to all teams and re-measure.
The report presents the result in one sentence, then the limits immediately after, then the recommendation. Samuel avoids saying the template "reduces findings", since the design cannot show cause; he says adopting services "had fewer findings". The recommendation to re-measure is framed as a way to test that more strongly. The report also includes the matched-pair table, so readers can judge the pairing themselves. A short appendix gives the kappa calculation and the blinded relabelling result.
Choosing pairs
Pairs are matched on number of containers, age of the service and team size, using records from the cloud team. Samuel writes the matching rules before looking at findings so the choice of pairs cannot be influenced by results. Two services have no good match and are excluded, which he reports. He also checks that both groups had similar numbers of deployments over the term, since more changes could mean more findings regardless of the template, and finds the counts close.
Sharing data
The matched-pair table and labels are kept with the analysis script so another student could rerun the comparison and check the result.
Ethics note
Findings are reported by service pair, never by person or team name.
Worked example
Dr Jensen asks whether reviewers knew which services used the template. They did; Samuel repeats the labelling on a sample with service names hidden and gets the same kappa, reducing concern about bias.
Synthetic case. Data are synthetic.
Errors and limits
Matching reduces some differences but not unmeasured ones.
Exercises
- EX097 — Design. How were groups compared? Rubric: 3 = 12 vs 12 matched pairs; 2 = partial; 1 = vague; 0 = none.
- EX098 — Results. What were medians and kappa? Rubric: 3 = 1 vs 4; kappa 0.82; 2 = partial; 1 = vague; 0 = none.
- EX099 — Blind check. How was reviewer bias checked? Rubric: 3 = relabel with names hidden; 2 = partial; 1 = vague; 0 = none.
Synthesis
Fair comparisons and reliable labels make findings credible.
Continue studying — chapter 33 of 40
← Chapter 32 (draft) · Course chapter index · Chapter 34 (draft) →