Chapter 4 — Baseline replication and pilot
Lesson map
Lesson 04 published. Map, example, EX022.
flowchart TD
N0["Learning outcomes"]
N1["4.1 Why replicate first"]
N2["Lab 1 — overlap baseline"]
N3["4.2 Reading overlap"]
N4["Lab 2 — calibration pilot"]
N5["4.3 Reading the pilot"]
N6["4.4 Feasibility decision"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX022"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["4.1 Why replicate first"]
N2["Lab 1 — overlap baseline"]
N3["4.2 Reading overlap"]
N4["Lab 2 — calibration pilot"]
N5["4.3 Reading the pilot"]
N6["4.4 Feasibility decision"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX022"]
[Learning outcomes]
|
v
[4.1 Why replicate first]
|
v
[Lab 1 — overlap baseline]
|
v
[4.2 Reading overlap]
|
v
[Lab 2 — calibration pilot]
|
v
[4.3 Reading the pilot]
|
v
[4.4 Feasibility decision]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX022]
Learning outcomes
You will (1) replicate a simple published-style baseline before building on it; (2) compute a Brier score and calibration bins; (3) compute set overlap between indicator feeds; (4) decide from a pilot whether the main study is feasible.
4.1 Why replicate first
Building on an unreplicated baseline compounds error. A pilot that reproduces a known, simple measurement — such as overlap between two indicator feeds, a quantity often discussed in sharing literature — checks that your pipeline, parsing and definitions behave before the expensive study.
Tomas's pilot uses two such checks: Jaccard overlap between two small synthetic indicator feeds, and a Brier score on twelve pilot judgments. Both quantities can be computed by hand, so a mismatch between the script and the hand calculation points to a bug in parsing or definitions, not to a property of the corpus.
Lab 1 — overlap baseline
"""Lab 1 (ch 04): replicate a baseline - Jaccard overlap of synthetic indicator sets from two feeds."""
feed_a = {"203.0.113.7", "203.0.113.9", "198.51.100.44", "bad.example", "c2.example"}
feed_b = {"203.0.113.9", "198.51.100.44", "c2.example", "drop.example"}
inter, union = feed_a & feed_b, feed_a | feed_b
print(f"|A|={len(feed_a)} |B|={len(feed_b)} |A&B|={len(inter)} |A|B|={len(union)}")
print(f"Jaccard={len(inter)/len(union):.3f}")
print("only in A:", sorted(feed_a - feed_b))
print("only in B:", sorted(feed_b - feed_a))
Run with python3 lab_overlap.py. Verified output:
|A|=5 |B|=4 |A&B|=3 |A|B|=6
Jaccard=0.500
only in A: ['203.0.113.7', 'bad.example']
only in B: ['drop.example']
4.2 Reading overlap
Three of six distinct indicators are shared, a Jaccard of 0.500. Overlap says nothing about quality: two feeds can agree on stale indicators. Bianco's Pyramid of Pain reminds analysts that hashes and IP addresses are cheap for adversaries to change, so high overlap on low-pyramid indicators is modest news.
For Tomas the overlap lab is a pipeline check, not a result: the 0.500 matches what he computes by hand from the six indicators, so the parsing and normalisation code behave as intended before the calibration work depends on them.
Lab 2 — calibration pilot
"""Lab 2 (ch 04): Brier score and calibration bins for synthetic analyst probability judgments."""
judgments = [ # (stated probability that the claim is true, outcome 1=true 0=false)
(0.9, 1), (0.9, 1), (0.9, 0), (0.7, 1), (0.7, 0), (0.7, 1),
(0.3, 0), (0.3, 1), (0.3, 0), (0.1, 0), (0.1, 0), (0.1, 0),
]
brier = sum((p - o) ** 2 for p, o in judgments) / len(judgments)
print(f"n={len(judgments)} Brier={brier:.3f}")
for p in sorted({p for p, _ in judgments}, reverse=True):
outs = [o for q, o in judgments if q == p]
print(f"stated {p:.1f} observed {sum(outs)/len(outs):.2f} (k={len(outs)})")
Run with python3 lab_calibration.py. Verified output:
n=12 Brier=0.183
stated 0.9 observed 0.67 (k=3)
stated 0.7 observed 0.67 (k=3)
stated 0.3 observed 0.33 (k=3)
stated 0.1 observed 0.00 (k=3)
4.3 Reading the pilot
Twelve pilot judgments give a Brier score of 0.183 (0 is perfect, 0.25 is what always saying 0.5 earns on balanced outcomes). The 0.9 bin is confirmed only two times in three, suggesting overconfidence at the top; the 0.1 bin is well behaved. With three items per bin, none of this is a finding — it is a test that the code works and a hint about where sample size must be largest.
Tomas checked the first four Brier terms by hand against the script and they matched. He wrote the 0.9-bin hint in his lab notebook as exploratory and asked for the main sample to over-weight high-confidence judgments, turning a three-item curiosity into a design decision recorded before the main data were coded.
4.4 Feasibility decision
The pilot passes if parsing succeeds, coders can resolve outcomes for at least the floor fraction, and the scoring code reproduces hand calculations. It fails if outcomes are mostly unresolvable or coding takes far longer than budgeted.
In Tomas's pilot the third test passed, because the four hand-computed Brier terms matched the script. The time test is the one candidates tend to skip: if coding one judgment takes three times the budget, 300 resolvable judgments become a different project, and the place to find that out is the pilot, not month nine.
Worked example
Tomas hand-computes the Brier score for the first four pilot items and matches the script. He records the 0.9-bin hint in his lab notebook as exploratory, not as a result, and requests that the main sample over-weight high-confidence judgments so that bin is estimated with adequate precision.
Synthetic case. Both feeds and all twelve pilot judgments are invented.
Errors and limits
Never publish pilot numbers as findings. Indicator values here use documentation addresses and .example names and are not real threats.
Exercises
- EX022 — Jaccard read. Quote the Jaccard value and list the indicators only in feed A. Rubric: 3 = 0.500; 203.0.113.7 and bad.example; 2 = value only; 1 = wrong; 0 = none.
- EX023 — Overlap meaning. Why does high overlap between two hash-based feeds not prove either is valuable? Rubric: 3 = both could be stale/low-pyramid; overlap ≠ quality or relevance; 2 = partial; 1 = it proves value; 0 = none.
- EX024 — Brier quote. What Brier score does the pilot report, and what is the reference value for always saying 0.5? Rubric: 3 = 0.183 and 0.25; 2 = one; 1 = wrong; 0 = none.
- EX025 — Top bin. What does the 0.9 bin suggest, and why is it not a finding? Rubric: 3 = possible overconfidence (0.67 observed); k=3 too small, pilot only; 2 = one half; 1 = proven overconfidence; 0 = none.
- EX026 — Hand check. Compute (p−o)² for the first pilot item (0.9, 1). Rubric: 3 = 0.01; 2 = method right, arithmetic slip; 1 = wrong method; 0 = none.
- EX027 — Sampling change. What sampling change does the pilot motivate? Rubric: 3 = over-sample high-confidence judgments for precision; 2 = bigger sample generally; 1 = vague; 0 = none.
- EX028 — Pilot fails. Name two observations that would make Tomas stop after the pilot. Rubric: 3 = two of mostly unresolvable outcomes / coding time far over budget / script mismatch with hand calc; 2 = one; 1 = low Brier; 0 = none.
Synthesis
Replicate the simple thing, verify code against hand calculation, treat pilot numbers as hints, and decide feasibility by pre-set rules.
Continue studying — chapter 04 of 08
← Chapter 03 (draft) · Course chapter index · Chapter 05 (draft) →