Chapter 12 — Probability and statistics

Lesson map

Lesson 12 published. Map, example, EX034.

flowchart TD
  N0["Learning outcomes"]
  N1["12.1 Sampling for audit"]
  N2["12.2 Confidence intervals"]
  N3["Lab — Audit sample and confidence interval"]
  N4["12.3 Common errors"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX034"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["12.1 Sampling for audit"]
  N2["12.2 Confidence intervals"]
  N3["Lab — Audit sample and confidence interval"]
  N4["12.3 Common errors"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX034"]

[Learning outcomes]
    |
    v
[12.1 Sampling for audit]
    |
    v
[12.2 Confidence intervals]
    |
    v
[Lab — Audit sample and confidence interval]
    |
    v
[12.3 Common errors]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX034]

Learning outcomes

You will (1) estimate a proportion from an audit sample; (2) compute and interpret a confidence interval; (3) avoid common statistical errors in security reports.

12.1 Sampling for audit

Fernway has about four thousand user and service accounts across systems. Hannah cannot review every account, so Lucas draws a random sample of two hundred and checks each for problems such as missing owners or stale access. He finds twenty-six with problems. The sample proportion is thirteen per cent. Random sampling matters: if he had checked only accounts he already suspected, the estimate would be biased upward. He records how the sample was drawn, with the random seed, so the audit can be repeated and checked by someone else.

12.2 Confidence intervals

A single proportion hides uncertainty. Lucas computes a ninety-five per cent confidence interval using the Wilson method, which behaves well for proportions. The lab below shows the estimate and interval, and the implied range of affected accounts across the population. He reports that between roughly nine and eighteen per cent of accounts likely have problems, about 360 to 740 accounts. That range is more honest and more useful than thirteen per cent alone, because it tells Hannah how much clean-up effort to plan for.

Lab — Audit sample and confidence interval

# Wilson 95% interval for an account-audit sample (synthetic).
import math
N, n, x, z = 4000, 200, 26, 1.96
p = x / n
den = 1 + z*z/n; centre = (p + z*z/(2*n)) / den
half = z * math.sqrt(p*(1-p)/n + z*z/(4*n*n)) / den
lo, hi = centre - half, centre + half
print(f"sample {x}/{n} = {p:.1%}; Wilson 95% CI {lo:.1%} to {hi:.1%}")
print(f"implied affected accounts: {lo*N:.0f} to {hi*N:.0f} of {N}")

Run with python3 lab_sample.py. Verified output:

sample 26/200 = 13.0%; Wilson 95% CI 9.0% to 18.4%
implied affected accounts: 361 to 735 of 4000

12.3 Common errors

Lucas lists errors he has seen in security reports: percentages without counts, comparisons of small numbers presented as dramatic changes, averages of skewed data and confusing correlation with cause. A report claiming phishing clicks fell by half after training, based on four clicks falling to two, says little. He resolves to always report counts with proportions, intervals where sampling is involved and the method used. Statistics do not make a report more scientific unless used carefully; misused, they lend false authority to weak claims.

Worked example

Lucas samples 200 of 4,000 accounts, finds 26 with problems, computes a Wilson interval and reports a range of affected accounts.

Synthetic case. Accounts and findings are synthetic.

Errors and limits

Intervals assume random sampling; convenience samples invalidate them. Problems found depend on reviewer consistency.

Exercises

• EX034 — Why random. Why must the audit sample be random?
  Show marking criteria

  Rubric: 3 = targeted samples bias the estimate;; 2 = bias;; 1 = faster;; 0 = none.
• EX035 — Lab interval. What interval did the lab report?
  Show marking criteria

  Rubric: 3 = about 9.0 to 18.4 per cent, roughly 361 to 735 accounts;; 2 = interval;; 1 = vague;; 0 = none.
• EX036 — Four to two. Why is phishing clicks fell by half, four to two, weak evidence?
  Show marking criteria

  Rubric: 3 = tiny counts make the change meaningless by chance;; 2 = small numbers;; 1 = strong;; 0 = none.

Synthesis

Sampling with stated methods and intervals turns partial audits into honest estimates.

Continue studying — chapter 12 of 40

← Chapter 11 (draft) · Course chapter index · Chapter 13 (draft) →