Chapter 5 — Main study and independent validation | Doctoral research Authorized offensive security and red team

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 5 — Main study and independent validation

Lesson map

Lesson 05 published. Map, example, EX027.

flowchart TD
  N0["Learning outcomes"]
  N1["5.1 Running to plan"]
  N2["5.2 Analysis"]
  N3["5.3 Independent validation"]
  N4["5.4 Confirmatory vs exploratory"]
  N5["5.4b Negative results"]
  N6["5.4c Sensitivity analyses"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX027"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["5.1 Running to plan"]
  N2["5.2 Analysis"]
  N3["5.3 Independent validation"]
  N4["5.4 Confirmatory vs exploratory"]
  N5["5.4b Negative results"]
  N6["5.4c Sensitivity analyses"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX027"]

[Learning outcomes]
    |
    v
[5.1 Running to plan]
    |
    v
[5.2 Analysis]
    |
    v
[5.3 Independent validation]
    |
    v
[5.4 Confirmatory vs exploratory]
    |
    v
[5.4b Negative results]
    |
    v
[5.4c Sensitivity analyses]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX027]

Learning outcomes

You will (1) run the main study to the pre-registered plan; (2) analyse primary and secondary measures with uncertainty; (3) commission independent validation; (4) separate confirmatory from exploratory findings; and (5) report failures and limits.

5.1 Running to plan

Randomise or alternate arms (prose vs checker) across matched engagement slots. Hold topology, briefing length and allowed tool categories fixed. Do not tune the checker using main-study outcomes.

Elena interleaves prose and checker slots across the term so time and lab load do not favour one arm. Topology images, briefing scripts and allowed tool lists are pinned. Checker tuning from main-study outcomes is forbidden; any bug fix needs an amendment. Hold the plan in a short checklist that operators see before each slot.

5.2 Analysis

Primary: difference in out-of-scope probes per hour with a confidence interval. Secondary: time-to-stop; false-stop rate; operator workload scores. Pre-specify any hierarchical model for operator effects.

"""Lab S: reporting attack-success rates honestly: Wilson intervals,
seed variation and a paired bootstrap. Synthetic trial outcomes only."""
import math, random
def wilson(k, n, z=1.96):
    p = k/n; d = 1 + z*z/n
    c = (p + z*z/(2*n))/d; h = z*math.sqrt(p*(1-p)/n + z*z/(4*n*n))/d
    return c - h, c + h
for k, n in ((3, 20), (30, 200), (300, 2000)):
    lo, hi = wilson(k, n); print(f"successes={k}/{n} rate={k/n:.3f} wilson95=[{lo:.3f}, {hi:.3f}]")
# five seeds of a synthetic defence evaluation: per-seed attack success rate
rates = []
for seed in range(5):
    r = random.Random(seed); rates.append(sum(r.random() < 0.12 for _ in range(200))/200)
m = sum(rates)/5; sd = math.sqrt(sum((x-m)**2 for x in rates)/4)
print("per-seed rates:", " ".join(f"{x:.3f}" for x in rates), f"mean={m:.3f} sd={sd:.3f}")
# paired bootstrap: baseline vs defence on the same 300 synthetic trials
r = random.Random(42)
base = [r.random() < 0.30 for _ in range(300)]
defn = [b and r.random() < 0.6 for b in base]   # defence blocks ~40% of successes
diffs = []
for _ in range(2000):
    idx = [r.randrange(300) for _ in range(300)]
    diffs.append(sum(base[i] for i in idx)/300 - sum(defn[i] for i in idx)/300)
diffs.sort()
print(f"baseline_asr={sum(base)/300:.3f} defended_asr={sum(defn)/300:.3f} "
      f"reduction_ci95=[{diffs[50]:.3f}, {diffs[1949]:.3f}]")

Expected output:

successes=3/20 rate=0.150 wilson95=[0.052, 0.360]
successes=30/200 rate=0.150 wilson95=[0.107, 0.206]
successes=300/2000 rate=0.150 wilson95=[0.135, 0.166]
per-seed rates: 0.110 0.130 0.135 0.105 0.145 mean=0.125 sd=0.017
baseline_asr=0.350 defended_asr=0.190 reduction_ci95=[0.120, 0.203]

Lab S habits apply: report counts and intervals, not a single p-value story. Small samples produce wide Wilson intervals, so Elena's claim rests on twenty engagements per arm and on the interval for the difference. Secondary measures are reported with the same discipline, and labelled secondary. The lab makes the point concrete: the same 0.150 rate gives a Wilson interval of 0.052 to 0.360 at 3/20, but 0.135 to 0.166 at 300/2000.

5.3 Independent validation

Ask researchers who did not build the checker to repeat the scoring of out-of-scope events from raw logs, or to re-run a subset of engagements. Independence catches optimistic grading.

Elena's validators are two researchers who did not build the checker. They re-score out-of-scope labels from redacted logs, blind to arm, and report percent agreement. Disagreements are adjudicated and counted. Agreement below a pre-set threshold triggers adjudication of the full set, not only the disagreements. Validators record each disagreement with the log line and both labels, so the adjudicator sees evidence rather than two competing opinions.

5.4 Confirmatory vs exploratory

Confirmatory results answer the pre-registered question. Exploratory analyses (for example, which deny-reasons dominate) are labelled as such and do not rescue a failed primary hypothesis.

Elena's confirmatory claim is the difference in out-of-scope probes per hour. That port-not-in-scope dominated denies is exploratory: useful for operators, not a rescue if the primary had failed. Keep the line bright in every table and figure caption, so a reader cannot mistake an exploratory plot for the primary result. In her results chapter, the deny-reason breakdown sits in a separate exploratory section placed after the primary rate table.

5.4b Negative results

A non-significant primary result is still a contribution if the study was powered and executed as planned. Publish the rates, intervals and limits. Negative results prevent other labs from repeating the same blind alley and keep the literature honest.

Elena writes the negative-result paragraph into her thesis template before the main study, so she is ready to publish either way. A powered null result still tells other labs that this checker, under these conditions, did not move the rate. State the rates, intervals and limits even when H1 is not supported.

5.4c Sensitivity analyses

Plan one or two sensitivity checks in advance: exclude the highest-volume operator; recompute with a stricter out-of-scope definition. If the qualitative conclusion moves, discuss why. Sensitivity is not a search for a significant p-value.

Elena plans two: exclude the highest-volume operator, and recompute with a stricter out-of-scope definition that treats any deny-listed host contact as out of scope even if the operator stopped at once. Both are named in the pre-registration. If either check moves the qualitative conclusion, discuss why in the limitations, rather than searching for a significant p-value.

5.5 Worked example (synthetic NHUSTL case)

Main study (20+20 engagements): prose 0.17/hour (interval 0.12–0.23); checker 0.03/hour (0.01–0.07); difference about 0.14/hour. Independent validators agreed on 98% of out-of-scope labels. Exploratory: most denies were port-not-in-scope. Failure mode: one engagement had a lab network renumbering that invalidated CIDRs mid-run — excluded per pre-registered rule, reported in limits.

The primary result supports H1 under the stated conditions. Independent agreement of 98% shows the labels were not optimistic grading by Elena alone. The excluded engagement is named in the results table and limitations, with the rule that justified it.

5.6 Errors and limits

Peeking and stopping early when results look good.
Validators who co-authored the tool.
Hiding exclusions.

Each error biases the result. Peeking and stopping early when results look good inflates false findings. Validators who co-authored the tool share the incentive to find an effect. Hiding exclusions lets the researcher shape the data without readers knowing. Check the analysis and validation plan for all three. Elena's analysis plan, written before collection, addresses all three. It fixes twenty engagements per arm, names validators outside the checker team, and lists every excluded engagement with its reason.

Exercises

EX027 — Design. How will you keep arms comparable? Rubric: 3 = topology, briefing, tools, timing; 2 = three; 1 = one; 0 = none.
EX028 — Primary result. Report Elena's primary result in one sentence. Rubric: 3 = rates, intervals, conditions; 2 = rates only; 1 = "it worked"; 0 = none.
EX029 — Independence. Design a validation protocol. Rubric: 3 = who, what, blinding, agreement metric; 2 = three; 1 = one; 0 = none.
EX030 — Exclusion. Was excluding the renumbered engagement correct? Rubric: 3 = matches pre-reg, reported; 2 = partial; 1 = hide it; 0 = none.
EX031 — Exploratory. Label one exploratory finding correctly. Rubric: 3 = labelled, not used as primary; 2 = partial; 1 = presented as confirmatory; 0 = none.
EX032 — Stats. What interval fact from the lab supports cautious claims from small n? Rubric: 3 = wide intervals at small n; 2 = partial; 1 = vague; 0 = wrong.

Synthesis

The main study follows the plan, quantifies uncertainty, seeks independent eyes and draws a bright line between confirmatory and exploratory claims.

Continue studying — chapter 05 of 08

← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →