Chapter 7 — Probability and statistics
Lesson map
Lesson 07 published. Map, example, EX019.
flowchart TD
N0["Learning outcomes"]
N1["7.1 Rare events and base rates"]
N2["7.2 Lookalike names"]
N3["Lab — Lab — name similarity"]
N4["7.3 Lab result"]
N5["7.4 Choosing a threshold"]
N6["7.5 Statistics for decisions"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX019"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["7.1 Rare events and base rates"]
N2["7.2 Lookalike names"]
N3["Lab — Lab — name similarity"]
N4["7.3 Lab result"]
N5["7.4 Choosing a threshold"]
N6["7.5 Statistics for decisions"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX019"]
[Learning outcomes]
|
v
[7.1 Rare events and base rates]
|
v
[7.2 Lookalike names]
|
v
[Lab — Lab — name similarity]
|
v
[7.3 Lab result]
|
v
[7.4 Choosing a threshold]
|
v
[7.5 Statistics for decisions]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX019]
Learning outcomes
You will (1) reason about rare events and false alarms; (2) compute and interpret similarity scores for package names; (3) choose thresholds for review; (4) understand base rates in security checks.
7.1 Rare events and base rates
Most new dependencies are legitimate. If 1 in 1,000 proposed packages were malicious and a check flagged 5% of legitimate ones, most flags would still be false alarms. This is the base-rate effect: alerts must be cheap to review or teams will ignore them.
Suppose 10,000 proposals arrive in a year: about 10 are malicious and about 500 legitimate ones are flagged. Even if every malicious one is caught, only about 2% of flags are real. The check is still worthwhile if reviewing a flag takes minutes, but not if each needs a long investigation, which is why Kwame designs the review to be quick.
7.2 Lookalike names
Ohm et al. (2020) documented malicious packages that used names close to popular ones (typosquatting). A defensive check compares each newly proposed dependency name with popular names and asks a human to look when they differ by a single character.
Attackers rely on hurried developers typing a name from memory or copying it from a forum post. The check cannot judge intent; it only notices similarity and passes the decision to a person who can look at the publisher, the package's age and its download history. Popular names are taken from the packages Coastline already uses, plus a list of widely used packages in the ecosystem.
Lab — Lab — name similarity
"""Flag new dependency names within edit distance 1 of a popular package (synthetic names)."""
def dist(a, b):
prev = list(range(len(b) + 1))
for i, ca in enumerate(a, 1):
cur = [i]
for j, cb in enumerate(b, 1):
cur.append(min(prev[j] + 1, cur[j - 1] + 1, prev[j - 1] + (ca != cb)))
prev = cur
return prev[-1]
popular = ["requestz", "escape-html", "qr-make", "fare-calc"]
proposed = ["escape-htm1", "qr-maker", "datefmt", "requests"]
for p in proposed:
near = [x for x in popular if 0 < dist(p, x) <= 1]
print(f"{p:12} {'REVIEW: near ' + ', '.join(near) if near else 'ok'}")
Run with python3 lab_typo.py. Verified output:
escape-htm1 REVIEW: near escape-html
qr-maker REVIEW: near qr-make
datefmt ok
requests REVIEW: near requestz
7.3 Lab result
Three of four proposed names are flagged: escape-htm1 (one character from escape-html), qr-maker (one from qr-make), and requests (one from requestz). datefmt is not near any popular name. The requests flag shows why review matters: in another ecosystem "requests" may be a well-known legitimate package, so the human reviewer checks publisher, history and download counts before deciding.
The three flags show different edits: escape-htm1 swaps a letter for a digit, qr-maker adds one letter, and requests replaces the final letter. Edit distance 1 means exactly one such insertion, deletion or substitution.
7.4 Choosing a threshold
A distance of 1 catches most single typing errors with few flags; a distance of 2 would flag many more legitimate names. Kwame measures over a month: 38 new dependency proposals, 4 flagged at distance 1, of which 1 was suspicious and blocked after review.
One suspicious package out of four flags is a precision of about 25%, which is acceptable for a review that takes a few minutes each. At distance 2, Kwame's trial produced far more flags for the same single catch, so the extra review time bought nothing. The month's counts justify the choice better than intuition would, and they will be checked again as the dependency list grows.
7.5 Statistics for decisions
Kwame reports the flag rate and the reviewed outcomes, not just "the check works". Over time these counts estimate the check's precision.
Counts also show what the check cannot do. One catch in a month is too few to estimate how many malicious packages it misses, since those are never seen. Kwame states this limit and suggests a periodic manual review of a sample of unflagged additions to estimate the miss rate. Decisions about tightening or loosening the check then rest on data rather than on the memory of a single dramatic catch.
Worked example
The suspicious package, a near-copy of qr-make published two days earlier by a new account, was never installed. Ingrid adds the check to the pull-request pipeline and asks for monthly counts.
Synthetic case. Package names and counts are synthetic.
Errors and limits
Similarity checks miss malicious packages with unrelated names and those introduced through legitimate maintainers' accounts.
Exercises
- EX019 — Base-rate effect. Why are most flags false alarms? Rubric: 3 = rare malicious packages; base rate; 2 = partial; 1 = vague; 0 = none.
- EX020 — Lab flags. Which names did the lab flag? Rubric: 3 = escape-htm1, qr-maker, requests; 2 = two; 1 = one; 0 = none.
- EX021 — Month results. What did the month of use show? Rubric: 3 = 38 proposals, 4 flagged, 1 blocked; 2 = partial; 1 = vague; 0 = none.
Synthesis
Similarity checks help when paired with human review and honest counts.
Continue studying — chapter 07 of 50
← Chapter 06 (draft) · Course chapter index · Chapter 08 (draft) →