◇ OpenCyber Academy
Master's · Chapter 6 · English

Chapter 6 — Data collection and analysis

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 6 — Data collection and analysis

Lesson map

Lesson 06 published. Map, example, EX033.

flowchart TD
  N0["Learning outcomes"]
  N1["6.1 Headline result"]
  N2["6.2 Within strata"]
  N3["6.3 Quality and safety"]
  N4["6.4 What remains uncertain"]
  N5["Lab — Bootstrap difference and stratified rates"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX033"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["6.1 Headline result"]
  N2["6.2 Within strata"]
  N3["6.3 Quality and safety"]
  N4["6.4 What remains uncertain"]
  N5["Lab — Bootstrap difference and stratified rates"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX033"]

[Learning outcomes]
    |
    v
[6.1 Headline result]
    |
    v
[6.2 Within strata]
    |
    v
[6.3 Quality and safety]
    |
    v
[6.4 What remains uncertain]
    |
    v
[Lab — Bootstrap difference and stratified rates]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX033]

Learning outcomes

You will (1) estimate a difference in proportions with a bootstrap interval; (2) stratify by a confounder; (3) interpret results against the pre-committed rule.

6.1 Headline result

After six months, Hana runs the analysis on 240 requests per period, here reproduced on synthetic data. On-time completion rises from 0.708 to 0.929, a difference of 0.221. A bootstrap with 4,000 resamples, following Efron and Tibshirani (1994), gives a 95% interval from 0.154 to 0.287. The interval excludes zero by a wide margin. But the request mix also changed: complex requests fell from 33% of the total to 17%, which is exactly the rival she predicted in Chapter 4, so the headline difference cannot be read as the portal's effect on its own.

6.2 Within strata

Hana therefore looks within groups. Simple requests improve from 0.800 to 0.960, a difference of 0.160. Complex requests improve from 0.525 to 0.780, a difference of 0.255. Both groups improve, so the portal effect is not just a change in mix, although part of the headline difference comes from the mix. She reports both the overall and stratified results in the same table, with the stratified figures given first, so that readers do not take the larger overall number as the effect of the portal and cannot quote it without seeing the qualification.

6.3 Quality and safety

The completeness review finds that 38 of 40 sampled answers met the Article 15 checklist after launch, compared with 35 of 40 before, so quality did not fall. No misdirected disclosures were recorded in either period. Staffing records show no change. Applying her pre-committed rule, Hana concludes that the portal counts as an improvement. She is careful to add that the review sample is small and that a single misdirected disclosure in future would change the assessment, so the safety outcome must keep being monitored after the dissertation ends and must be owned by someone at Brookfield.

6.4 What remains uncertain

Hana lists what her data cannot show. The study covers one company and one year; the portal launched alongside a website redesign that may have pushed customers toward online channels; and the bootstrap interval reflects sampling variation, not the design's weaknesses. She writes that the portal is associated with a substantial improvement within both simple and complex requests, and that the most likely explanation is the portal, with the website redesign as a remaining rival. Dr. Pritchard praises the sentence because it says exactly as much as the evidence allows.

Lab — Bootstrap difference and stratified rates

# Before/after on-time rates with a bootstrap 95% interval, overall and within simple vs complex requests.
import random
r = random.Random(9)
def period(n, p_simple, p_complex, share_complex):
    out = []
    for _ in range(n):
        cx = r.random() < share_complex
        out.append((cx, r.random() < (p_complex if cx else p_simple)))
    return out
before = period(240, 0.82, 0.55, 0.30); after = period(240, 0.95, 0.70, 0.15)
rate = lambda s: sum(ok for _, ok in s) / len(s)
def boot(a, b, k=4000):
    d = sorted(rate(r.choices(b, k=len(b))) - rate(r.choices(a, k=len(a))) for _ in range(k))
    return d[int(0.025 * k)], d[int(0.975 * k)]
lo, hi = boot(before, after)
print(f"overall on-time {rate(before):.3f} -> {rate(after):.3f}; difference {rate(after)-rate(before):.3f} (95% CI {lo:.3f} to {hi:.3f})")
print(f"complex share {sum(c for c, _ in before)/240:.2f} -> {sum(c for c, _ in after)/240:.2f}")
for cx, name in [(False, "simple"), (True, "complex")]:
    a = [x for x in before if x[0] == cx]; b = [x for x in after if x[0] == cx]
    print(f"{name:7s} {rate(a):.3f} -> {rate(b):.3f}; difference {rate(b)-rate(a):.3f}")

Run with python3 lab_diff.py. Verified output:

overall on-time 0.708 -> 0.929; difference 0.221 (95% CI 0.154 to 0.287)
complex share 0.33 -> 0.17
simple  0.800 -> 0.960; difference 0.160
complex 0.525 -> 0.780; difference 0.255

Worked example

Hana reports on-time completion rising from 0.708 to 0.929 (difference 0.221, 95% interval 0.154 to 0.287), shows improvement within simple (0.160) and complex (0.255) requests, and confirms that completeness and safety did not worsen.

Synthetic case. All rates are synthetic; Efron and Tibshirani (1994) is real.

Errors and limits

Bootstrap intervals assume independent requests and do not cover design bias. Stratifying by one confounder leaves others.

Exercises

Synthesis

Chapter 6 produces a bounded, stratified result. Chapter 7 writes it up with honest limitations.

Continue studying — chapter 06 of 08

← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →