Chapter 7 — Probability and statistics
Lesson map
Lesson 07 published. Map, example, EX019.
flowchart TD
N0["Learning outcomes"]
N1["7.1 Distributions"]
N2["Lab — Lab — alert latency"]
N3["7.2 Lab result"]
N4["7.3 Objectives"]
N5["7.4 Sampling"]
N6["7.5 Probability of events"]
N7["Choosing the objective"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX019"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["7.1 Distributions"]
N2["Lab — Lab — alert latency"]
N3["7.2 Lab result"]
N4["7.3 Objectives"]
N5["7.4 Sampling"]
N6["7.5 Probability of events"]
N7["Choosing the objective"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX019"]
[Learning outcomes]
|
v
[7.1 Distributions]
|
v
[Lab — Lab — alert latency]
|
v
[7.2 Lab result]
|
v
[7.3 Objectives]
|
v
[7.4 Sampling]
|
v
[7.5 Probability of events]
|
v
[Choosing the objective]
|
v
[Worked example] --> [Exercise EX019]
Learning outcomes
You will (1) summarise operational measurements with appropriate statistics; (2) understand skewed distributions; (3) use percentiles for service objectives; (4) avoid misleading averages.
7.1 Distributions
Many cloud measurements, such as response times and alert delays, are skewed: most are small, a few are very large. The mean is pulled by large values; the median describes the typical case; high percentiles describe the worst common cases.
An average response time of 300 ms could hide that most requests take 100 ms while a few take several seconds. Users who hit those slow requests have a poor experience, and for alerts, a slow delivery could mean a late response to an incident. Reporting median and high percentiles together gives a truer picture than any single number.
Lab — Lab — alert latency
"""Summarise synthetic alert latencies (minutes): mean, median, 95th percentile (stdlib only)."""
import statistics
lat = [2, 3, 3, 4, 4, 5, 5, 5, 6, 6, 7, 7, 8, 9, 10, 12, 15, 21, 38, 95]
q = statistics.quantiles(lat, n=20, method="inclusive")
print(f"n {len(lat)}, mean {statistics.mean(lat):.1f}, median {statistics.median(lat)}, p95 {q[18]:.1f}, max {max(lat)}")
Run with python3 lab_latency.py. Verified output:
n 20, mean 13.2, median 6.5, p95 40.9, max 95
7.2 Lab result
Twenty synthetic alert latencies have mean 13.2 minutes, median 6.5, 95th percentile 40.9 and maximum 95. The mean is twice the median because of a few long delays; the p95 shows that about one alert in twenty takes over 40 minutes, which matters for incidents such as a leaked key.
The maximum of 95 minutes is one alert that took over an hour and a half to reach anyone, which in a key-leak scenario could mean the key was used in that time. The mean alone would suggest a moderate delay; the p95 shows the real risk lies in the tail. Elena follows up the slowest cases to find out why they took so long.
7.3 Objectives
Solent sets an objective that 95% of critical alerts reach the on-call engineer within 15 minutes. The current p95 of 40.9 minutes misses it, and the long tail becomes an engineering problem to solve.
An objective turns a vague wish into a measurable target. Fifteen minutes is chosen because a leaked key or exposed service can be abused quickly once found. Missing the target by a wide margin shows the alert path needs redesign, such as paging directly rather than through a shared mailbox, and the same measure later shows whether the fix worked.
7.4 Sampling
Twenty values are a small sample; percentiles from small samples are uncertain. Elena collects a month of data before drawing conclusions.
With twenty values, the 95th percentile rests on just one or two observations, so a single unusual alert can move it dramatically. A month gives hundreds of values and a much steadier estimate. Elena reports the sample size alongside each percentile, so readers know how much weight to place on it. She also checks whether the month was typical, since a holiday period could make alert traffic unusually light.
7.5 Probability of events
If a control fails independently with probability 0.01 per day, the chance of at least one failure in a year is 1 − 0.99^365, about 97%. Rare daily events become near certain over time.
The calculation uses the complement: the chance of no failures in a year is 0.99 multiplied by itself 365 times, about 2.6%, so at least one failure is about 97% likely. This explains why systems that 'almost never' fail still fail regularly at scale. It argues for monitoring every control and designing for recovery, not assuming success.
Choosing the objective
The 15-minute objective is chosen with the on-call team, based on how fast a leaked key must be revoked to limit harm. Objectives set without the people who must meet them tend to be ignored. Elena records the reasoning beside the objective so it can be revisited when the alert system changes. She also proposes measuring it monthly and reporting misses with their causes, not just the percentage.
Worked example
Looking at the longest delays, Elena finds they all come from alerts routed through a shared email list checked only in working hours. Routing critical alerts to the paging system is expected to remove the tail.
Synthetic case. Latencies are synthetic.
Errors and limits
Percentiles from small samples are unstable; report sample size.
Exercises
- EX019 — Lab stats. What were mean, median and p95? Rubric: 3 = 13.2, 6.5, 40.9; 2 = two; 1 = one; 0 = none.
- EX020 — Yearly chance. What is the chance of at least one failure in a year at 0.01/day? Rubric: 3 = about 97%; 2 = partial; 1 = vague; 0 = none.
- EX021 — Tail cause. What caused the long delays? Rubric: 3 = email list checked in working hours; 2 = partial; 1 = vague; 0 = none.
Synthesis
Use medians and percentiles for skewed operational data.
Continue studying — chapter 07 of 50
← Chapter 06 (draft) · Course chapter index · Chapter 08 (draft) →