Chapter 1 — Advanced research integrity and methods
Lesson map
Lesson 01 published. Map, example, EX001.
flowchart TD
N0["Learning outcomes"]
N1["1.1 The case"]
N2["1.2 Defining an error"]
N3["Lab — Coder agreement"]
N4["1.3 Reading agreement"]
N5["1.4 Integrity and data handling"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX001"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["1.1 The case"]
N2["1.2 Defining an error"]
N3["Lab — Coder agreement"]
N4["1.3 Reading agreement"]
N5["1.4 Integrity and data handling"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX001"]
[Learning outcomes]
|
v
[1.1 The case]
|
v
[1.2 Defining an error]
|
v
[Lab — Coder agreement]
|
v
[1.3 Reading agreement]
|
v
[1.4 Integrity and data handling]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX001]
Learning outcomes
You will (1) frame a doctoral question about firewall rule-base quality; (2) design valid measurement of configuration errors; (3) establish ethical and data-handling rules for partner configurations.
1.1 The case
Ines Carvalho is a doctoral candidate at the fictional Halden University, supervised by Prof. Marcus Lindqvist. Her partner organisations are three fictional regional bodies, led by Westbrook Water, whose firewalls have grown over fifteen years to hundreds of rules each. Engineers suspect many rules are redundant, overly broad or contradict earlier rules, but nobody can say how many or whether reviews help. Ines's thesis asks whether rule-base complexity predicts configuration errors and whether a structured review procedure reduces them. Wool (2004) found such a relationship in a study of real firewall configurations; her work aims to test it on modern rule bases and add an intervention.
1.2 Defining an error
A study of errors needs a definition that two people would apply the same way. Ines uses four categories drawn from the firewall literature and NIST SP 800-41 Rev. 1: a rule that is fine; a rule shadowed by an earlier rule, so it never matches; a rule broader than its documented purpose, such as allowing any source; and a rule with no traffic hits for a defined period and no documented owner. Each category has written decision rules and examples. The definitions are fixed before any partner data are coded, so results cannot reshape them.
Lab — Coder agreement
"""Cohen's kappa for two coders labelling 40 synthetic firewall rules as ok / shadowed / permissive / unused (stdlib only)."""
from collections import Counter
a = list("OOOOOOOOOOOOOOOOSSSSSSPPPPPPPPUUUUUUUUUU")
b = list("OOOOOOOOOOOOOOOSSSSSSOPPPPPPPUUUUUUUUUUP")
n = len(a); po = sum(x == y for x, y in zip(a, b)) / n
ca, cb = Counter(a), Counter(b); pe = sum(ca[k] * cb[k] for k in set(a) | set(b)) / n**2
k = (po - pe) / (1 - pe)
print(f"rules {n}; observed agreement {po:.3f}; chance {pe:.3f}; kappa {k:.3f}")
print("disagreements:", [(i + 1, x, y) for i, (x, y) in enumerate(zip(a, b)) if x != y])
Run with python3 lab_kappa.py. Verified output:
rules 40; observed agreement 0.900; chance 0.285; kappa 0.860
disagreements: [(16, 'O', 'S'), (22, 'S', 'O'), (30, 'P', 'U'), (40, 'U', 'P')]
1.3 Reading agreement
Two trained coders independently label forty synthetic rules. They agree on 36, an observed agreement of 0.90, and Cohen's kappa, which corrects for agreement expected by chance, is 0.86. Landis and Koch (1977) would call this almost perfect agreement, though Ines treats such labels as rough guides. The four disagreements matter more than the number: two are between fine and shadowed, where the coders read an object group differently, and two between permissive and unused. Each leads to a sharper decision rule in the codebook, and the agreement exercise is repeated on fresh rules after the change.
1.4 Integrity and data handling
Firewall configurations reveal how a partner's network is built, which makes them sensitive. Ines's data agreement allows her to receive configurations only after the partner replaces real addresses and names with consistent pseudonyms, keeps the files on an encrypted university server and forbids publication of any rule text. All examples in her thesis and in this book are synthetic. Her ethics approval covers interviews with engineers, who can withdraw at any time. She never probes the partners' networks; every observation comes from configurations and logs the partners choose to share.
Worked example
Ines writes the research question, the four-category codebook and the data agreement, then runs the forty-rule agreement exercise. Kappa of 0.86 and four clear disagreement patterns lead to two codebook clarifications before real coding begins.
Synthetic case. Ines Carvalho, Halden University, Westbrook Water and all figures and rules are synthetic.
Errors and limits
Kappa summarises agreement but hides where disagreement lies; always inspect disagreements, and remember that high agreement does not guarantee the categories are the right ones.
Exercises
- EX001 — Error categories. Define the four rule categories Ines uses and give one decision rule for each. Rubric: 3 = all four with clear decision rules;; 2 = four categories without rules;; 1 = fewer than four;; 0 = none.
- EX002 — Kappa reading. Interpret observed agreement 0.90 and kappa 0.86 for a committee. Rubric: 3 = explains chance correction and limits;; 2 = explains kappa;; 1 = restates numbers;; 0 = none.
- EX003 — Disagreement value. Why are the four disagreements more useful than the kappa value? Rubric: 3 = show where the codebook is ambiguous and lead to fixes;; 2 = mention ambiguity;; 1 = vague;; 0 = none.
- EX004 — Pseudonymisation. Explain why partners pseudonymise configurations before sharing. Rubric: 3 = network design sensitivity, consistent mapping preserves analysis;; 2 = sensitivity;; 1 = vague;; 0 = none.
- EX005 — Fixing definitions first. What bias does fixing definitions before coding prevent? Rubric: 3 = post-hoc redefinition to fit results;; 2 = bias;; 1 = vague;; 0 = none.
- EX006 — Observation boundary. Why does Ines never probe partner networks? Rubric: 3 = authorisation, ethics, risk to operations;; 2 = not allowed;; 1 = vague;; 0 = none.
- EX007 — Prior work link. How does Wool (2004) shape the research question? Rubric: 3 = prior finding of complexity-error link to test and extend;; 2 = cites;; 1 = vague;; 0 = none.
Synthesis
A defensible doctoral study of rule-base errors starts with a precise codebook, measured coder agreement and strict handling of sensitive configurations.