Chapter 6 — Data collection and analysis
Lesson map
Lesson 06 published. Map, example, EX034.
flowchart TD
N0["Learning outcomes and connection"]
N1["6.1 Collection"]
N2["6.2 Coding failures"]
N3["6.3 Difference-in-differences"]
N4["6.4 Testbed results"]
N5["6.5 Validation on Fenwick College"]
N6["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX034"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["6.1 Collection"]
N2["6.2 Coding failures"]
N3["6.3 Difference-in-differences"]
N4["6.4 Testbed results"]
N5["6.5 Validation on Fenwick College"]
N6["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX034"]
[Learning outcomes and connection]
|
v
[6.1 Collection]
|
v
[6.2 Coding failures]
|
v
[6.3 Difference-in-differences]
|
v
[6.4 Testbed results]
|
v
[6.5 Validation on Fenwick College]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX034]
Learning outcomes and connection
You will (1) collect data according to a frozen protocol; (2) code failure logs reliably and measure agreement; (3) estimate a difference-in-differences effect; (4) analyse testbed results; and (5) validate on a second site. It executes the proposal of Chapter 5 and supplies the results Chapter 7 will write up.
6.1 Collection
Eight weeks of LK-HS-1 data are collected under the amended protocol: four weeks before and four weeks after the staggered rollout, with half of managed browsers (randomized by device identifier) receiving the hybrid policy first. Freeze the analysis script and record its hash before unblinding the cohort labels.
Keep a collection log noting outages or gaps in the proxy and browser telemetry, with dates, because a missing day in one cohort would bias the comparison. Check before unblinding that the randomization put roughly equal numbers of devices in each cohort and that both used the proxy path in similar proportions.
6.2 Coding failures
Each failed series has a log excerpt. Two coders (C1, C2) assign one of four labels — size (proxy reset after first segment), cert (certificate validation), timeout, other — using a written codebook, independently.
# Lab 6 (synthetic): agreement between two coders labelling 60 failure logs, then the share of
# failures attributed to "split ClientHello mishandled" after reconciliation.
from collections import Counter
labels = ["size", "cert", "timeout", "other"]
a = ["size"]*30 + ["cert"]*12 + ["timeout"]*10 + ["other"]*8
b = ["size"]*27 + ["timeout"]*3 + ["cert"]*11 + ["other"] + ["timeout"]*9 + ["size"] + ["other"]*8
n = len(a); po = sum(x == y for x, y in zip(a, b)) / n
ca, cb = Counter(a), Counter(b); pe = sum(ca[l]*cb[l] for l in labels) / n**2
print(f"observed agreement {po:.3f} chance {pe:.3f} Cohen's kappa {(po-pe)/(1-pe):.3f}")
Expected output:
observed agreement 0.917 chance 0.323 Cohen's kappa 0.877
A κ of 0.88 indicates strong agreement beyond chance; disagreements (mostly size vs timeout, because a dropped second segment can look like a timeout) are resolved by discussion and logged.
Keep the codebook with an example log excerpt for each label, especially for the difficult boundary between size and timeout. Coders work from excerpts without cohort labels, so that knowing whether the hybrid group was offered cannot influence the label they choose.
6.3 Difference-in-differences
With a concurrent control cohort, the effect is the change in the treated cohort minus the change in the control cohort, estimated separately by path. Synthetic results: proxy path +20.4 per 10,000 (95% CI 15.1–25.8); direct path +0.6 (−1.0 to 2.2). H1 is supported. The control cohort's own small rise on the proxy path (+0.6, Exercise EX034) captures seasonality, rival 4, which the before/after design alone would have attributed to the hybrid group.
Report the estimate with its interval and the number of series behind each cell, so the calculation can be checked. The direct-path interval includes zero, which supports the second part of H1. The control cohort's rise shows why the concurrent control matters: without it, seasonal change would have been counted as an effect of the hybrid group.
6.4 Testbed results
In the isolated testbed copy of the proxy, 200 handshakes per condition: one-segment hello without codepoint 0/200 failures; one-segment hello with the hybrid codepoint 0/200; two-segment hello without the codepoint (padding only) 187/200; two-segment hello with codepoint 189/200. Failures track segmentation, not the codepoint: H2 supported, rival 3 rejected for this proxy model. After applying the vendor's firmware update in the testbed, two-segment failures fall to 0/200, which anticipates H3.
The four conditions vary size and codepoint independently, which is what lets the testbed separate the two rivals the logs could not. Report the testbed's firmware versions before and after the update and the padding used for each condition, so another lab can reproduce the result with the same proxy model.
6.5 Validation on Fenwick College
Fenwick (synthetic) has a different proxy vendor. Applying the frozen analysis: proxy-path excess +1.2 per 10,000 (−0.8 to 3.2) — no meaningful effect. The mechanism is product-specific; the dissertation must say "the legacy proxy model at Larkspur", not "inspecting proxies". A null result at the second site is a finding about scope, not a failure of the study.
Freeze the analysis script before opening Fenwick data, as for Larkspur, and record its hash. Report Fenwick's interval as well as its estimate: it includes zero and excludes an effect as large as Larkspur's. Recording the proxy vendor and version at Fenwick lets readers see which products the null result covers.
6.6 Common errors and limits
- Unblinding cohort labels before freezing the script.
- Treating κ as a pass/fail without examining which categories disagree.
- Generalizing a product-specific defect to a category of products.
- Synthetic numbers here illustrate methods; they are not measurements of any product.
Each error has a guard in this chapter: a script hash recorded before unblinding, a look at which labels disagreed rather than a κ threshold alone, the Fenwick null result used to limit the claim to one proxy model, and a clear statement that the numbers are synthetic.
Exercises — chapter 6
- EX032 — Freeze record. Write the freeze record (what, hash, witness, date). Rubric: 3 = verifiable.
- EX033 — Kappa interpretation. Recompute Lab 6 after changing three size labels in coder B to timeout; interpret. Rubric: 3 = correct κ and comment on which confusion matters.
- EX034 — DiD by hand. Treated proxy path 5.0→26.0, control proxy path 5.1→5.7 per 10,000: compute the DiD. Rubric: 3 = 20.4 and explanation.
- EX035 — Testbed table. Explain how the four testbed conditions discriminate between rivals 1 and 3. Rubric: 3 = correct logic.
- EX036 — Scope from validation. Write the scope sentence implied by the Fenwick null result. Rubric: 3 = product-specific claim, no overgeneralization.
- EX037 — Threats. List three threats to the DiD's parallel-trends assumption here. Rubric: 3 = specific to the setting.
Chapter synthesis
Frozen scripts, double coding with κ, a difference-in-differences estimate by path, a testbed that varies size and codepoint independently, and validation at a second site turn a correlation into a scoped causal account.
Continue studying — chapter 06 of 08
← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →