Chapter 4 — Baseline replication and pilot
Lesson map
Lesson 04 published. Map, example, EX021.
flowchart TD
N0["Learning outcomes and connection"]
N1["4.1 Terms"]
N2["4.2 Diagnosing a failed reproduction: InvBench-22"]
N3["4.3 Verifying the instrument"]
N4["4.4 Practical activity — Lab 4: verifying the classi"]
N5["4.5 Tuning without leakage"]
N6["4.6 A bounded pilot"]
N7["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX021"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["4.1 Terms"]
N2["4.2 Diagnosing a failed reproduction: InvBench-22"]
N3["4.3 Verifying the instrument"]
N4["4.4 Practical activity — Lab 4: verifying the classi"]
N5["4.5 Tuning without leakage"]
N6["4.6 A bounded pilot"]
N7["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX021"]
[Learning outcomes and connection]
|
v
[4.1 Terms]
|
v
[4.2 Diagnosing a failed reproduction: InvBench-22]
|
v
[4.3 Verifying the instrument]
|
v
[4.4 Practical activity — Lab 4: verifying the classi]
|
v
[4.5 Tuning without leakage]
|
v
[4.6 A bounded pilot]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX021]
Learning outcomes and connection
After this chapter you can (1) distinguish repeatability, reproducibility and replicability; (2) diagnose why a published inventory result fails to reproduce; (3) verify an instrument against known inputs before trusting its output; (4) tune parameters without leaking test data; and (5) run a bounded pilot that ends in an explicit go/no-go decision. It applies the proposal of Chapter 3 before the main study of Chapter 5.
4.1 Terms
Following the ACM's artifact review and badging vocabulary: repeatability — the same team obtains the same result with the same setup; reproducibility — a different team obtains it with the original artefacts; replicability — a different team obtains it with independently developed artefacts. A doctoral baseline should at least be reproduced, and the reproduction log should record every deviation, including versions of scanners, parsers and the TLS group registry used to classify codepoints.
In the Ciphertrail Study the InvBench-22 work is a reproduction, because it reuses the released corpus.
4.2 Diagnosing a failed reproduction: InvBench-22
The synthetic paper InvBench-22 reports that its scanner found "97% of quantum-vulnerable assets". The candidate obtains the released corpus and gets 61%. Three causes emerge, in order of discovery. First, duplicate counting: the paper counted findings, not uses — the same server appeared under a short name, a fully qualified name and two IP addresses, each counted as a hit against a single ground-truth item. Second, ground-truth leakage: the ground truth was built partly from the scanner's own earlier output, so items the scanner could not see were missing from the denominator. Third, registry drift: the paper's classifier predated the final ML-KEM codepoints and labelled the obsolete pre-standard hybrid codepoint as "unknown", which the authors then excluded. None of these is fraud; each is a measurement decision that was not reported. The reproduction log states each cause, quantifies its effect and notes which are inferences rather than confirmed by the authors.
4.3 Verifying the instrument
Before measuring HRHN, verify that the passive classifier maps each TLS supported-group codepoint correctly. The IANA TLS Supported Groups registry lists the codepoints; RFC 10024 defines X25519MLKEM768, SecP256r1MLKEM768 and SecP384r1MLKEM1024. Unit tests with known inputs are the cheapest validity evidence a thesis can have. Two design rules matter. An unknown codepoint must be classified as unknown, never as safe. And identity must be canonicalized before counting, so that one use is one row.
4.4 Practical activity — Lab 4: verifying the classifier and counting uses
# Lab 4 (synthetic): verify the passive classifier and show how duplicate counting inflates recall.
# TLS supported-group codepoints (IANA registry): 0x001D x25519, 0x0017 secp256r1,
# 0x11EC X25519MLKEM768, 0x11EB SecP256r1MLKEM768, 0x11ED SecP384r1MLKEM1024,
# 0x6399 X25519Kyber768Draft00 (obsolete pre-standard codepoint).
GROUPS = {0x001D: "classical", 0x0017: "classical", 0x11EC: "hybrid", 0x11EB: "hybrid",
0x11ED: "hybrid", 0x6399: "pre-standard hybrid"}
def classify(code):
return GROUPS.get(code, "unknown") # never default an unknown codepoint to "safe"
tests = [(0x001D, "classical"), (0x11EC, "hybrid"), (0x6399, "pre-standard hybrid"), (0xFFFF, "unknown")]
assert all(classify(c) == want for c, want in tests), "classifier unit test failed"
print("classifier unit tests: 4/4 pass")
# Synthetic handshake findings: (server pseudonym, port, selected group)
findings = [("srv-014", 443, 0x001D), ("srv-014", 443, 0x001D), ("SRV-014.halden.example", 443, 0x001D),
("srv-022", 8443, 0x11EC), ("srv-031", 443, 0x0017), ("srv-031", 443, 0x0017)]
truth_vulnerable = {("srv-014", 443), ("srv-031", 443), ("srv-040", 22)} # adjudicated uses
def canon(host):
return host.lower().split(".")[0]
raw_hits = sum(1 for h, p, g in findings if classify(g) == "classical")
uses = {(canon(h), p) for h, p, g in findings if classify(g) == "classical"}
print(f"raw classical findings: {raw_hits} -> naive 'recall' {raw_hits}/{len(truth_vulnerable)} = {raw_hits/len(truth_vulnerable):.2f}")
print(f"canonical uses found: {len(uses & truth_vulnerable)} -> recall {len(uses & truth_vulnerable)/len(truth_vulnerable):.2f}")
Expected output:
classifier unit tests: 4/4 pass
raw classical findings: 5 -> naive 'recall' 5/3 = 1.67
canonical uses found: 2 -> recall 0.67
A "recall" above 1 is the clearest possible signal of a unit error, but subtler inflation — say from 0.61 to 0.97 — looks plausible and survives review unless the counting unit is stated. The missing SSH use on srv-040 also illustrates the observer model: the TLS sensor cannot find it.
The lab separates two checks that are easy to confuse. The unit tests show the classifier maps known codepoints correctly, including an unknown one. The counting step shows how the same server, logged repeatedly and under two names, can inflate recall. Keep both in the instrument's test suite and run them before every collection, recording the IANA registry retrieval date beside the results.
4.5 Tuning without leakage
The fusion rule has parameters: how long a passive observation remains valid, how hostnames are matched to code repositories, and the confidence threshold for attributing a negotiated group to an asset. Tune them on a development split (in the Ciphertrail Study, HRHN hospital 1), freeze them, and evaluate on the held-out hospitals 2 and 3. Record every tuning run, including those that were abandoned; a garden of forking paths that is hidden is indistinguishable from p-hacking.
Keep a tuning log with the date, the parameter values and the result on hospital 1 for every run. When the rules are frozen, record the configuration hash in the same log.
4.6 A bounded pilot
The pilot runs for two weeks on one HRHN segment with 140 services. Pre-stated go/no-go criteria: classifier unit tests all pass; pseudonymization verified on a sample of 200 records by a second person; no operational incident attributable to the study; adjudicator agreement (Cohen's κ) at least 0.70 on a 60-item calibration set; and the protocol takes no more than four engineer-hours per 100 services. The synthetic pilot meets four criteria but records κ = 0.62, because adjudicators disagreed on whether a configured-but-never-negotiated RSA key-exchange cipher suite is a "use". The decision is go with amendment: the codebook now defines a use as any algorithm reachable by a peer in negotiation, with a separate flag for "configured, not observed".
4.7 Common errors and limits
- Counting findings rather than uses; always state the unit.
- Building ground truth from the tool under test.
- Using an outdated codepoint registry; record its retrieval date.
- Tuning on test data, even informally by "looking at a few results".
- A pilot without pre-stated criteria cannot fail and therefore teaches little.
Each error has a guard in this study: canonical identifiers for uses, adjudication blind to the tool, a dated registry, a split by hospital with frozen rules, and pre-stated pilot criteria. The pilot's low agreement of 0.62 is an example of a criterion doing its job; it forced a codebook amendment before the main study rather than after.
Exercises — chapter 4
- EX021 — Vocabulary. Classify three scenarios as repeatability, reproducibility or replicability. Rubric: 3 = all correct with a one-line justification each.
- EX022 — Reproduction log. Write the log entry for InvBench-22 cause 2, separating observation from inference. Rubric: 3 = effect quantified, inference labelled.
- EX023 — Extend the classifier. Add unit tests to Lab 4 for SecP384r1MLKEM1024 (0x11ED, already in the table) and for an unknown codepoint 0x0200. Rubric: 3 = tests pass and unknown stays unknown.
- EX024 — Canonical identity. Propose a canonicalization rule for a use that handles IP addresses, aliases and load balancers. Rubric: 3 = addresses at least one failure mode of the rule.
- EX025 — Split design. Design development/test splits for HRHN that prevent leakage when two hospitals share a PKI. Rubric: 3 = identifies shared-PKI leakage and mitigates it.
- EX026 — Go/no-go. Given κ = 0.62 and all other criteria met, write the decision memo. Rubric: 3 = decision justified, amendment specific, re-test criterion stated.
Chapter synthesis
Baselines are reproduced before they are beaten; instruments are verified on known inputs; the counting unit is stated; tuning is separated from testing; and a pilot exists to be able to fail. InvBench-22's inflated recall came from three unreported measurement decisions, not from the scanner.
Continue studying — chapter 04 of 08
← Chapter 03 (draft) · Course chapter index · Chapter 05 (draft) →