Chapter 42 — Applied research on response-system limits
Lesson map
Lesson 42 published. Map, example, EX083.
flowchart TD
N0["Learning outcomes and connection"]
N1["42.1 Limit claims"]
N2["42.2 Experiment design"]
N3["42.4 External validity"]
N4["42.5 Reproducibility"]
N5["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX083"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["42.1 Limit claims"]
N2["42.2 Experiment design"]
N3["42.4 External validity"]
N4["42.5 Reproducibility"]
N5["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX083"]
[Learning outcomes and connection]
|
v
[42.1 Limit claims]
|
v
[42.2 Experiment design]
|
v
[42.4 External validity]
|
v
[42.5 Reproducibility]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX083]
Learning outcomes and connection
After this chapter you can (1) formulate a testable claim about where a designed response system stops supporting reliable conclusions; (2) design a reproducible experiment with a baseline and controlled conditions; (3) separate analytic, simulated and empirical evidence; and (4) state external-validity limits. It builds on Chapters 7, 29, 36 and 40 and prepares Chapters 46, 47 and 49. Research here serves design decisions: the output is a measured limit the system's operators and owners can use, not a publication.
42.1 Limit claims
A limit claim has the form: "Under conditions C, property P of the system holds while parameter X stays within range R, and fails beyond it." Example: "With independent loss of telemetry, case classification based on 2-of-3 corroborating signals stays at or above 90% accuracy while the missing rate per signal is at most about 20%." The claim is useful because it can be monitored: if a source's missing rate exceeds the limit, the operators know conclusions have become unreliable. Lab 42 below tests exactly this example and places the limit at 0.196.
42.2 Experiment design
Elements: baseline (system behaviour with complete telemetry); manipulated variable (missing rate, delay, volume); measured outcome (classification accuracy against known ground truth); controls (everything else fixed); replication (several seeds or runs); analysis plan (written before the run). Synthetic data give known ground truth; recorded sanitised data give realism but uncertain labels. Use both where possible and report each separately. Simulation models need verification (does the code do what the model says?) and validation (does the model represent the real system?) (Sargent, 2013).
42.3 Worked example (synthetic): case quality under missing telemetry
"""Lab 42 (synthetic): where does case classification stop being reliable as telemetry goes missing?
A case is correctly classified if at least 2 of its 3 corroborating signals arrive. Seeds are fixed."""
import random, math
def trial(missing, n=400, seed=0):
rng = random.Random(seed); ok = 0
for _ in range(n):
got = sum(1 for _ in range(3) if rng.random() > missing)
ok += got >= 2
return ok / n
print("missing accuracy(mean of 5 seeds) min..max analytic")
for m in [0.0, 0.1, 0.2, 0.3, 0.4, 0.5]:
acc = [trial(m, seed=s) for s in range(5)]
q = 1 - m; analytic = q**3 + 3 * q**2 * (1 - q)
print(f" {m:.1f} {sum(acc)/5:.3f} {min(acc):.3f}..{max(acc):.3f} {analytic:.3f}")
target = 0.90
lo, hi = 0.0, 1.0
for _ in range(40):
mid = (lo + hi) / 2; q = 1 - mid
if q**3 + 3*q**2*(1-q) >= target: lo = mid
else: hi = mid
print(f"limit: accuracy >= {target} holds while missing rate <= {lo:.3f} (independent-loss assumption)")
Expected output:
missing accuracy(mean of 5 seeds) min..max analytic
0.0 1.000 1.000..1.000 1.000
0.1 0.967 0.945..0.985 0.972
0.2 0.888 0.863..0.912 0.896
0.3 0.779 0.757..0.800 0.784
0.4 0.649 0.625..0.677 0.648
0.5 0.512 0.490..0.545 0.500
limit: accuracy >= 0.9 holds while missing rate <= 0.196 (independent-loss assumption)
The simulated means track the analytic curve $q^3 + 3q^2(1-q)$ with $q = 1 - m$, which verifies the simulation code against the model. Accuracy stays above 0.9 while the missing rate is at most about 0.196. The spread across seeds (for example 0.863–0.912 at $m = 0.2$ with 400 cases per run) shows that a single run near the limit could fall on either side — a reason to report the limit with uncertainty and to monitor with margin, say alerting at 15% missing. Each point in the table rests on 5 seeds of 400 cases, so 2,000 simulated cases per missing rate.
42.4 External validity
The result rests on independent losses. Real losses are correlated: one collector outage removes all three signals from a site at once (Chapter 3). Under fully correlated loss, accuracy is simply $1 - m$, and the 90% limit falls to 10% missing. The experiment's next step is therefore a validation run on HVSS staging with a recorded collector-outage pattern, comparing accuracy under realistic correlation. The limit claim must state which assumption it rests on; operators should monitor correlated gaps (site-level silence) separately from random missingness.
42.5 Reproducibility
Publish within the organisation: code, parameters, seeds, the analytic check, raw results and the analysis plan. Another engineer should reproduce the table exactly. Pre-specifying the analysis protects against picking the most favourable presentation (Nosek et al., 2018).
For Lab 42, that means publishing the five seeds 0–4, the 400 cases per run and the analytic formula $q^3 + 3q^2(1-q)$. The bisection that produced 0.196 and the plan written before the run should be published with them. Another HVSS engineer can then rerun the code and reproduce the table digit for digit.
42.6 Errors and limits
- Limit claims without stated conditions.
- Simulation without verification against an analytic or reference case.
- Ignoring correlation structure.
- Single-run results presented as the limit.
The Lab 42 result illustrates how each of these errors would distort it. Stated as "telemetry loss up to 20% is fine", it drops the independence condition on which it rests. Without the analytic column, a bug in trial could pass unnoticed. Under fully correlated loss the 90% limit falls from about 19.6% to 10% missing. A single seed at m = 0.2 could have reported anything from 0.863 to 0.912, on either side of the target.
Exercises — chapter 42
EX083 — Correlated loss. Objective: test the assumption. Inputs: Lab 42. Steps: add a mode in which with probability $m$ all three signals are lost together (otherwise all arrive); compute accuracy for $m$ = 0.1 and 0.2 and the 90% limit. Expected evidence: code and output. Rubric: 3 = accuracy ≈ 0.9 and 0.8, limit 0.10, with interpretation; 2 = output only; 1 = attempt; 0 = none.
EX084 — Limit claim. Objective: formulate and plan. Inputs: the delay effect of Lab 36. Steps: write a limit claim about correlation-rule reliability versus event delay and design an experiment (baseline, manipulated variable, outcome, controls, replication, analysis plan). Expected evidence: claim and design. Rubric: 3 = testable claim with conditions and a reproducible design; 2 = partial; 1 = vague; 0 = none.
Chapter synthesis
Applied research in engineering finds the limits of a designed system: testable claims with conditions, controlled and replicated experiments, simulations verified against analysis, explicit external-validity assumptions and results operators can monitor.
Continue studying — chapter 42 of 50
← Chapter 41 · Course chapter index · Chapter 43 →
Continue to the next authored lesson.