Chapter 6 — Data collection and analysis
Lesson map
Lesson 06 published. Map, example, EX033.
flowchart TD
N0["Learning outcomes and connection"]
N1["6.1 Collection discipline"]
N2["6.2 Lab 6 — sensitivity to orifice coefficient"]
N3["6.3 Honest results paragraph"]
N4["6.5 From numbers to claims"]
N5["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX033"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["6.1 Collection discipline"]
N2["6.2 Lab 6 — sensitivity to orifice coefficient"]
N3["6.3 Honest results paragraph"]
N4["6.5 From numbers to claims"]
N5["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX033"]
[Learning outcomes and connection]
|
v
[6.1 Collection discipline]
|
v
[6.2 Lab 6 — sensitivity to orifice coefficient]
|
v
[6.3 Honest results paragraph]
|
v
[6.5 From numbers to claims]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX033]
Learning outcomes and connection
After this chapter you can (1) collect model and rig data under a frozen plan, (2) use paired seeds and bootstrap intervals, (3) run sensitivity analyses, and (4) write an honest results paragraph.
6.1 Collection discipline
Freeze thresholds from tuning seeds; run evaluation seeds; then Rig R2 with the same frozen values. Log seed, detector, attack class, alarm time, overfill at alarm. No peeking at R2 attack data before freezing.
Practically, write a small manifest containing k, h, tuning seed range and a hash of the analysis script, and refuse to load R2 attack files until that manifest exists. If you change k after seeing R2, you no longer have a validation — you have another tuning exercise, and you must say so. Keep a lab notebook with date, person, run IDs and anomalies (power blip, sensor recalibration); anomalies that coincide with alarms are rival explanations.
6.2 Lab 6 — sensitivity to orifice coefficient
# Lab 6: bootstrap CI for delay difference (paired) + sensitivity to orifice coefficient.
import random
def delay(seed, orifice=0.02, k=0.8, steps=900, onset=300):
rng=random.Random(seed); true=est=50.0; s=0.0
# keep equilibrium near 50: nominal inflow = orifice*50
for t in range(steps):
inflow=orifice*50 + rng.gauss(0,0.05)+(0.05 if t>=onset else 0)
true+=inflow-orifice*true
lvl=(50.0 if t>=onset else true)+rng.gauss(0,0.3)
est+=inflow+rng.gauss(0,0.02)-orifice*est
s=max(0.0,s+abs(lvl-est)-k)
if s>6 and t>=onset: return t-onset
return None
for orifice in (0.015,0.02,0.025):
d8=[]; d6=[]; diffs=[]
for i in range(150):
a=delay(i,orifice,0.8); b=delay(i,orifice,0.6)
if a is not None: d8.append(a)
if b is not None: d6.append(b)
if a is not None and b is not None: diffs.append(a-b)
rng=random.Random(1); boots=sorted(sum(rng.choice(diffs) for _ in diffs)/len(diffs) for _ in range(1000)) if diffs else []
print(f"orifice={orifice} detected_k0.8={len(d8)}/150 detected_k0.6={len(d6)}/150 "
f"mean_diff={sum(diffs)/len(diffs):.1f} 95%CI=[{boots[25]:.1f},{boots[974]:.1f}]")
Expected output:
orifice=0.015 detected_k0.8=150/150 detected_k0.6=150/150 mean_diff=6.0 95%CI=[5.6,6.4]
orifice=0.02 detected_k0.8=150/150 detected_k0.6=150/150 mean_diff=7.1 95%CI=[6.6,7.6]
orifice=0.025 detected_k0.8=150/150 detected_k0.6=150/150 mean_diff=8.6 95%CI=[7.9,9.4]
The delay cost of choosing k = 0.8 over 0.6 grows as the orifice coefficient rises (faster natural dynamics / different signal). Intervals describe simulation noise only.
The lab pairs seeds: each seed is run with both allowances, and the difference is taken within the seed, which removes noise that both runs share. The bootstrap then resamples those paired differences. All three orifice settings detect every episode at both allowances, so the trade-off is purely in delay. The intervals describe only simulation noise; they say nothing about how a real tank differs from the model.
6.3 Honest results paragraph
"In the synthetic Harbor Loop model, residual CUSUM detected 80/80 class A replay episodes (mean delay 37.7 steps, mean overfill at alarm 1.38) with 0/80 false-alarm runs; the ±1.5 band detected 0/80 with 11/80 false-alarm runs. Class B (flow meter also spoofed) evaded CUSUM in 80/80 trials. Delay cost of the more robust allowance k = 0.8 versus 0.6 was 6.0–8.6 steps across orifice coefficients 0.015–0.025. Rig R2 validation is reported separately with frozen thresholds."
The paragraph reports the failure in the same sentence pattern as the success, so class B cannot be skimmed past. It gives every figure with its denominator and names the model as synthetic. It also points forward to the rig results without claiming them. Check that each number matches the lab output exactly before it goes into the dissertation.
6.4 Common errors and limits
Independent seeds for paired comparisons; intervals presented as real-world uncertainty; hiding class B.
Paired comparisons need the same seed for both settings; independent seeds add noise that hides small differences. Bootstrap intervals describe simulation variability under the model, not uncertainty about real plants. Class B belongs in the main results table, not in an appendix, because it is the measured boundary of the method and an operator needs to know it. Report the seeds used for each comparison so others can repeat the pairing exactly.
6.5 From numbers to claims
After Labs 3 and 6 you have detection counts, delays, false-alarm runs and a sensitivity curve. Turning them into claims requires the ladder from the doctoral companion: a protocol observation is not a process inference. Say "CUSUM alarmed at mean delay 37.7 steps while true overfill was 1.38" (observation + process inference) rather than "CUSUM prevents overflows" (consequence claim without protection-layer evidence). When Rig R2 results arrive, report them in the same units so a reader can see transportability or its failure without hunting.
Keep a short table that pairs each number with the claim it supports and the level of that claim. A delay in steps supports a detection claim; overfill at alarm supports a process inference within the model; neither supports a statement about protection, because the study did not measure a protection layer. That table later feeds the claim register.
Exercises — chapter 6
- EX033 — 150-word results from Lab 3 and Lab 6. Run Lab 3 and Lab 6; write a 150-word results paragraph with numbers. Rubric: 3 = ~150 words with Lab 3/6 numbers for H1/H4 and bias/allowance; 2 = numbers without structure; 1 = short; 0 = none.
- EX034 — Independent seeds CI width for k=0.6. Rerun Lab 6 with independent seeds for k = 0.6; compare CI width. Rubric: 3 = wider CI under independent seeds vs paired, with direction; 2 = direction only; 1 = wrong; 0 = none.
- EX035 — Six uncertainty sources classified. Classify six uncertainty sources as stochastic, parametric or structural. Rubric: 3 = stochastic/parametric/structural with examples; 2 = classes only; 1 = one class; 0 = none.
- EX036 — Freeze rule before R2 validation. Write the freeze rule that must hold before R2 validation. Rubric: 3 = freeze k/h hashed before R2 attack data + invalidation rule; 2 = freeze without invalidation; 1 = vague; 0 = none.
- EX037 — Summarize never-alarm runs. Propose how to summarize runs that never alarm (right-censoring). Rubric: 3 = detection proportion + median among detected or survival-style summary; 2 = one better summary; 1 = mean with cap; 0 = none.
- EX038 — What Lab 6 intervals do not mean. State what Lab 6's intervals do not mean. Rubric: 3 = intervals ≠ plant uncertainty / structural error; 2 = one caveat; 1 = treats as plant truth; 0 = none.
Chapter synthesis
Frozen plans, paired designs, sensitivity sweeps and honest paragraphs — including the failure condition — are the analysis standard for this dissertation.
Continue studying — chapter 06 of 08
← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →