Chapter 4 — Baseline replication and pilot | Doctoral research Authorized offensive security and red team

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 04 published. Map, example, EX021.

flowchart TD
  N0["Learning outcomes"]
  N1["4.1 Baseline"]
  N2["4.2 Pilot of the intervention"]
  N3["4.3 Estimation"]
  N4["4.4 Instrument refinement"]
  N5["4.5 Go / no-go"]
  N6["4.5b Operator briefing"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX021"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["4.1 Baseline"]
  N2["4.2 Pilot of the intervention"]
  N3["4.3 Estimation"]
  N4["4.4 Instrument refinement"]
  N5["4.5 Go / no-go"]
  N6["4.5b Operator briefing"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX021"]

[Learning outcomes]
    |
    v
[4.1 Baseline]
    |
    v
[4.2 Pilot of the intervention]
    |
    v
[4.3 Estimation]
    |
    v
[4.4 Instrument refinement]
    |
    v
[4.5 Go / no-go]
    |
    v
[4.5b Operator briefing]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX021]

Learning outcomes

You will (1) replicate a baseline method under prose ROE; (2) run a pilot of the intervention; (3) estimate rates and variances; (4) refine instruments without peeking at final conclusions; and (5) decide go / no-go for the main study.

4.1 Baseline

Run the current practice — prose ROE read at the start — across a fixed lab topology. Log every probe attempt with host, port, action and timestamp. Score out-of-scope using the same predicate rules you will later enforce live, so the baseline is comparable.

Elena runs ten baseline engagements on the fixed NHUSTL topology. Scoring the baseline with the same predicates she will later enforce live means both arms use one definition of out-of-scope, so a difference can be attributed to the checker rather than to a change in scoring.

4.2 Pilot of the intervention

Enable the checker that evaluates proposals before execution. Record allows, denies and reasons. Watch for false stops that frustrate operators without improving safety.

Elena's pilot runs ten engagements with the checker on. She logs every allow and deny with its reason, and notes whether a deny was a false stop, such as a mistyped host that was actually in scope. False stops are a secondary measure: too many mean the checker is unusable, even if out-of-scope probes fall. In the pilot, both false stops came from one operator mistyping an in-scope host, which points to an interface fix rather than a policy change.

4.3 Estimation

Compute rates per hour with intervals. Note operator effects (some people probe more). Keep the pilot analysis descriptive; do not "succeed" the hypothesis yet.

"""Rebuild a synthetic engagement timeline from event records."""
events = [
    ("09:00", "ROE brief signed", "auth"),
    ("09:15", "scoped host inventory loaded (12 hosts)", "prep"),
    ("10:02", "service enumeration on 10.20.1.0/28", "enum"),
    ("10:45", "finding SYN-001 noted on staging UI", "find"),
    ("11:10", "stop: host 10.20.0.1 probed — out of scope", "stop"),
    ("11:12", "escalated to client contact; confirmed deny-list", "comm"),
    ("13:00", "resume on in-scope hosts only", "auth"),
    ("15:30", "debrief draft shared with client", "report"),
]
# print chronological with phase tags; compute stop latency
print("phase timeline:")
for t, msg, phase in events:
    print(f"  {t} [{phase:6}] {msg}")
t0 = 10*60+45; t1 = 11*60+10
print(f"minutes from SYN-001 note to stop: {t1-t0}")
print(f"stop reason present: yes")
print(f"events={len(events)} phases={len(set(p for _,_,p in events))}")

Expected output:

phase timeline:
  09:00 [auth  ] ROE brief signed
  09:15 [prep  ] scoped host inventory loaded (12 hosts)
  10:02 [enum  ] service enumeration on 10.20.1.0/28
  10:45 [find  ] finding SYN-001 noted on staging UI
  11:10 [stop  ] stop: host 10.20.0.1 probed — out of scope
  11:12 [comm  ] escalated to client contact; confirmed deny-list
  13:00 [auth  ] resume on in-scope hosts only
  15:30 [report] debrief draft shared with client
minutes from SYN-001 note to stop: 25
stop reason present: yes
events=8 phases=7

Elena reports pilot rates with intervals and does not test H1 yet. Operator effects are noted so the main study can plan for them, for example by modelling operator as a factor or by balancing slots across people. Keep the pilot analysis in a separate notebook from the main study. The synthetic timeline lab gives a concrete check: 25 minutes passed between the SYN-001 note and the stop, a latency the main study should track.

4.4 Instrument refinement

Clarify ROE wording, fix checker bugs, improve logging. Lock instruments before the main study. Document every change.

Elena's typo UX fix and the locked checker version are both recorded in the instrument log with dates. After lock, further changes need a supervisor-approved amendment. Locking stops silent drift between early and late engagements, and the main study does not start until the new version is pinned. Operators receive the locked version number in the briefing, so they can report if a different build appears.

4.5 Go / no-go

Go if: authorization still valid; instruments stable; pilot shows non-zero out-of-scope events under prose ROE (so there is something to reduce); ethics unchanged. No-go or redesign if the checker is unusable or the lab is unstable.

Elena's go decision lists all four. A no-go example is a false-stop rate above about 30%, which would force a UX redesign and a re-pilot before the main study. Going ahead with an unusable checker would produce data that measures frustration rather than scope compliance.

4.5b Operator briefing

Brief operators identically in both arms: same ROE text length, same topology map, same allowed tool categories. Differences should come from the checker, not from unequal preparation. Record briefing duration and attendance.

Elena's briefing script is identical in both arms and lasts the same number of minutes. Attendance is recorded so missing briefings can be reported as a limitation if they cluster in one arm. Unequal preparation would confound the comparison, so differences should come from the checker alone. The briefing pack also shows the same scoped inventory of twelve hosts in both arms, so neither group starts with a clearer map.

4.6 Worked example (synthetic NHUSTL case)

Baseline (10 engagements): 0.18 out-of-scope probes/hour (interval roughly 0.11–0.28). Pilot with checker (10 engagements): 0.04/hour; two false stops when an operator mistyped a host in scope. Timeline lab habits — note finding, stop, escalate — are rehearsed; minutes from SYN-001 note to stop in the synthetic timeline is 25. Elena fixes the typo UX and locks the checker version.

The baseline shows there is something to reduce: a non-zero out-of-scope rate under prose ROE. The pilot shows the checker can cut that rate, and exposes a UX bug that would have frustrated operators in the main study. Locking the checker after the fix keeps the main study comparable across engagements.

4.7 Errors and limits

Changing the scoring rules between baseline and pilot.
Treating pilot results as confirmatory.
Skipping false-stop analysis.

Each error damages the comparison. Changing scoring rules between baseline and pilot means the two rates are not measuring the same thing. Treating pilot results as confirmatory inflates the chance of a false finding. Skipping false-stop analysis leaves a UX problem that will confound the main study. Check the pilot report for all three before the go decision. Elena's go criteria list each one.

Exercises

EX021 — Baseline log. Specify the log fields. Rubric: 3 = host, port, action, time, decision, operator; 2 = four; 1 = two; 0 = none.
EX022 — Comparability. Why score baseline with the same predicates? Rubric: 3 = fair comparison; 2 = partial; 1 = vague; 0 = wrong.
EX023 — Timeline lab. What does the stop latency show? Rubric: 3 = 25 minutes, process existence; 2 = number; 1 = vague; 0 = wrong.
EX024 — False stop. How should Elena handle the mistyped-host stops? Rubric: 3 = UX fix, not weaken scope; 2 = partial; 1 = disable checker; 0 = none.
EX025 — Go criteria. Write four go criteria. Rubric: 3 = four apt; 2 = three; 1 = one; 0 = none.
EX026 — No-go. Give a no-go scenario and redesign. Rubric: 3 = scenario and redesign; 2 = scenario; 1 = vague; 0 = none.

Synthesis

Baseline and pilot produce comparable rates, expose instrument problems and support a disciplined go / no-go — not a premature claim of success.

Continue studying — chapter 04 of 08

← Chapter 03 (draft) · Course chapter index · Chapter 05 (draft) →