Chapter 6 — Data collection and analysis
Lesson map
Lesson 06 published. Map, example, EX033.
flowchart TD
N0["Learning outcomes"]
N1["6.1 Collecting data"]
N2["Lab — Sign-in success"]
N3["6.2 Reading the result"]
N4["6.3 Examining the failures"]
N5["6.4 The difference-in-differences"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX033"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["6.1 Collecting data"]
N2["Lab — Sign-in success"]
N3["6.2 Reading the result"]
N4["6.3 Examining the failures"]
N5["6.4 The difference-in-differences"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX033"]
[Learning outcomes]
|
v
[6.1 Collecting data]
|
v
[Lab — Sign-in success]
|
v
[6.2 Reading the result]
|
v
[6.3 Examining the failures]
|
v
[6.4 The difference-in-differences]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX033]
Learning outcomes
You will (1) collect aggregate data across the rollout; (2) estimate sign-in success with intervals; (3) examine the cases where passkeys failed.
6.1 Collecting data
Over three months, Halyard enrols seven departments in turn. Each week, the identity team exports aggregates: sign-in attempts and successes by method and department, reset and recovery tickets, and headcount. Jonah checks every file on arrival for completeness, comparing totals with the previous week and flagging gaps. Two weeks of one department's data are missing after a log storage fault; he records the gap and handles it as his analysis plan specified, by excluding those weeks from both groups rather than estimating values that were never measured.
Lab — Sign-in success
"""Sign-in success rate with a Wilson 95% interval for synthetic pilot and rollout groups (stdlib only)."""
import math
def wilson(k, n, z=1.96):
p = k/n; c = (p + z*z/(2*n)) / (1 + z*z/n); h = z*math.sqrt(p*(1-p)/n + z*z/(4*n*n)) / (1 + z*z/n)
return p, c-h, c+h
for name, k, n in [("password+OTP", 9412, 9640), ("passkey", 9581, 9655)]:
p, lo, hi = wilson(k, n); print(f"{name:13} success={k}/{n} rate={p:.3f} 95%CI=({lo:.3f}, {hi:.3f}) fails={n-k}")
Run with python3 lab_success.py. Verified output:
password+OTP success=9412/9640 rate=0.976 95%CI=(0.973, 0.979) fails=228
passkey success=9581/9655 rate=0.992 95%CI=(0.990, 0.994) fails=74
6.2 Reading the result
In the synthetic data, password plus one-time code sign-ins succeed 97.6 percent of the time, with a 95 percent interval of 97.3 to 97.9, while passkey sign-ins succeed 99.2 percent, interval 99.0 to 99.4. The intervals do not overlap, so passkeys are very likely more reliable for these staff. Failures fall from 228 to 74 in similar numbers of attempts. Jonah uses Wilson intervals because they behave well when rates are close to one. He reports the counts beside the percentages, so readers can judge the precision for themselves.
6.3 Examining the failures
The seventy-four passkey failures deserve attention. Jonah groups them by cause using the outcome codes: most are biometric retries that exceed a limit, some are devices without a registered passkey being used for the first time, and a handful are on an older browser that does not support the standard. None involves an attacker. Each cause suggests a fix: guidance on registering more than one device and a browser upgrade for the remaining machines. Examining failures turns an aggregate rate into practical improvements that Halyard can act on.
6.4 The difference-in-differences
The main analysis compares changes. In enrolled departments, the reset rate falls by about 2.2 per hundred staff per week; in not-yet-enrolled departments over the same weeks it falls by about 0.2. The difference, about 2.0, is Jonah's estimate of the effect attributable to passkeys, under the assumption that both groups would otherwise have followed similar trends. He checks that assumption by comparing their trends before any enrolment and finds them close. He reports the estimate with an interval and states the assumption openly.
Worked example
Passkey sign-in success is 99.2 percent against 97.6 percent for password plus code, with non-overlapping intervals; failures are grouped by cause and the difference-in-differences estimate is about 2.0 fewer resets per hundred staff per week.
Synthetic case. Departments, logs and results are synthetic.
Errors and limits
Filling missing weeks with guessed values hides the gap and can bias both groups.
Exercises
- EX033 — Success rates. What are the two sign-in success rates and intervals? Rubric: 3 = 97.6% (97.3–97.9) and 99.2% (99.0–99.4);; 2 = rates;; 1 = one;; 0 = none.
- EX034 — Failure counts. How many failures does each method have in the lab? Rubric: 3 = 228 and 74;; 2 = both;; 1 = one;; 0 = none.
- EX035 — Wilson choice. Why use Wilson intervals here? Rubric: 3 = they behave well for rates near one;; 2 = near 1;; 1 = simpler;; 0 = none.
- EX036 — Missing weeks. How are the two missing weeks handled? Rubric: 3 = excluded from both groups per plan, not imputed;; 2 = excluded;; 1 = estimated;; 0 = none.
- EX037 — Failure causes. Name two causes of passkey failures. Rubric: 3 = biometric retry limit, unregistered device, old browser (two);; 2 = one;; 1 = attacker;; 0 = none.
- EX038 — DiD estimate. What is the difference-in-differences estimate? Rubric: 3 = about 2.0 fewer resets per hundred staff per week;; 2 = 2.0;; 1 = 2.2;; 0 = none.
Synthesis
Careful collection, intervals around rates, cause analysis of failures and a comparison of changes give an honest picture of the effect.
Continue studying — chapter 06 of 08
← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →