Chapter 5 — Proposal and pilot
Lesson map
Lesson 05 published. Map, example, EX027.
flowchart TD
N0["Learning outcomes"]
N1["5.1 Proposal contents"]
N2["5.2 Templates"]
N3["5.3 Pilot protocol"]
N4["5.4 Instrument lock"]
N5["5.5 Go / no-go"]
N6["5.5b Early-signal discipline"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX027"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["5.1 Proposal contents"]
N2["5.2 Templates"]
N3["5.3 Pilot protocol"]
N4["5.4 Instrument lock"]
N5["5.5 Go / no-go"]
N6["5.5b Early-signal discipline"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX027"]
[Learning outcomes]
|
v
[5.1 Proposal contents]
|
v
[5.2 Templates]
|
v
[5.3 Pilot protocol]
|
v
[5.4 Instrument lock]
|
v
[5.5 Go / no-go]
|
v
[5.5b Early-signal discipline]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX027]
Learning outcomes
You will (1) write a short proposal your supervisor can approve; (2) build report templates for both arms; (3) run a pilot on synthetic findings; (4) estimate completion rates and tag-audit effort; and (5) lock instruments before the main collection.
5.1 Proposal contents
Include question, rationale, design, measures, ethics, timeline and risks. Attach authorization for synthetic assessments and consent forms. State dual-use limits in one paragraph your supervisor can quote.
For Noah, the question line reads: on CBM's synthetic segments, do ATT&CK-tagged findings reach a higher 30-day remediation rate than narrative-only findings? The design paragraph names between-findings randomisation on VLAN 40 with strata for severity and asset class. The risks section lists the planning sketch from chapter 3: if narrative completion sits near 40%, detecting a rise to 60% needs on the order of 100 findings per arm, so a slow assessment calendar is itself a risk to the timeline.
5.2 Templates
Narrative template fields: title, asset, evidence reference, impact, recommendation. Tagged template: the same plus ATT&CK technique IDs and parent tactic. Both templates explicitly forbid exploit steps or credentials.
Keeping every other field identical is what makes the comparison fair: if the tagged template also had a longer impact section, any difference in remediation could come from that extra text rather than from the ATT&CK fields. Noah therefore builds the tagged template by adding fields to the narrative one, never by rewriting it. The pinned ATT&CK version goes into the template footer, so a remediator opening a ticket months later sees the same technique definitions the writer used.
5.3 Pilot protocol
Generate 40 synthetic findings from a training assessment on VLAN 40; randomise; deliver tickets; observe a 14-day early signal if 30 days exceeds the term calendar; audit tags for correctness.
Forty findings is far below the roughly 100 per arm needed for the main comparison, so the pilot is not meant to answer the question. Its job is to show that randomisation produces balanced arms, that tickets arrive in CBM's system with the format flag recorded, that remediators can see the tag fields, and that the tag audit catches incorrect technique IDs. The 14-day signal is only an early check on variance for refining the sample-size sketch, never a preliminary result.
5.4 Instrument lock
Freeze templates, ATT&CK version, remediation definition and randomisation code. Record versions in the methods chapter. After lock, changes require a supervisor-approved amendment.
Locking matters most for the remediation definition. If Noah decided after seeing pilot tickets that a ticket closed as risk accepted counts as remediated, he could move the 30-day rate in either arm. With the lock in place, such a change becomes an amendment his supervisor must approve and that the methods chapter must report, so examiners can see whether any rule moved after data arrived.
5.5 Go / no-go
Go if authorization holds, templates are usable in the ticket UI, and format is recorded per ticket. No-go or redesign if remediators cannot see tags without opening a hidden field.
The hidden-field condition is not cosmetic. If remediators must click into a collapsed panel to see the technique IDs, many tagged tickets would in practice be read as narrative tickets, and the study would compare two arms that look the same to the people doing the work. A no-go here costs weeks; going ahead would cost the validity of the whole dissertation.
"""Check proposed engagement targets against a written scope (synthetic)."""
SCOPE_CIDRS = [("10.20.0.0", 16), ("192.168.50.0", 24)] # inclusive synthetic ranges
SCOPE_PORTS = {22, 80, 443, 8080}
DENY_HOSTS = {"10.20.0.1", "192.168.50.1"} # out-of-scope infra
def ip_int(s):
a,b,c,d = map(int, s.split("."))
return (a<<24)|(b<<16)|(c<<8)|d
def in_cidr(ip, net, bits):
mask = (0xFFFFFFFF << (32-bits)) & 0xFFFFFFFF
return (ip_int(ip) & mask) == (ip_int(net) & mask)
def check(host, port):
reasons = []
if host in DENY_HOSTS:
reasons.append("deny-list")
if not any(in_cidr(host, n, b) for n,b in SCOPE_CIDRS):
reasons.append("outside-cidr")
if port not in SCOPE_PORTS:
reasons.append("port-not-in-scope")
return ("ALLOW" if not reasons else "DENY"), reasons
tests = [
("10.20.1.10", 443),
("10.20.0.1", 443),
("192.168.50.20", 22),
("192.168.50.20", 3389),
("8.8.8.8", 80),
("192.168.1.10", 80),
]
for h,p in tests:
v, r = check(h,p)
print(f"target={h}:{p} decision={v} reasons={','.join(r) or '-'}")
Expected output:
target=10.20.1.10:443 decision=ALLOW reasons=-
target=10.20.0.1:443 decision=DENY reasons=deny-list
target=192.168.50.20:22 decision=ALLOW reasons=-
target=192.168.50.20:3389 decision=DENY reasons=port-not-in-scope
target=8.8.8.8:80 decision=DENY reasons=outside-cidr
target=192.168.1.10:80 decision=DENY reasons=outside-cidr
5.5b Early-signal discipline
A 14-day early signal can inform logistics but must not rewrite the primary 30-day measure. If you report it, label it exploratory. The dissertation's confirmatory claim remains the pre-registered 30-day completion comparison.
In Noah's pilot, the 14-day signal of 10 of 20 tagged against 6 of 20 narrative early responses is used only to refine the sample-size sketch. It is never reported as a preliminary result. If the main study's 30-day result differs, the confirmatory claim stays with the 30-day result. Keep the exploratory label in every table.
5.6 Worked example (synthetic CBM case)
Pilot early signal: tagged 10/20 early responses versus narrative 6/20. Tag audit: 2/20 tags wrong — Noah adds a second-person tag check. Scope lab rules remain in the assessment runbook: deny-listed hosts and out-of-scope ports never enter study data.
The pilot met its purpose: randomisation, delivery and the format flag worked, and the tag audit found a problem early. Two wrong tags in twenty is a 10% error rate, enough to blur a difference between arms, so Noah adds a second-person check before the main collection. The early-response figures inform logistics, not conclusions.
5.7 Errors and limits
- Pilots on production.
- Changing templates mid-main-study.
- Skipping tag audit.
Each error damages the main study. Pilots on production systems break the authorization and expose real data. Changing templates during the main study means the two halves of the data measure different things, and the instrument lock exists to prevent it. Skipping the tag audit leaves wrong technique IDs undetected, as the pilot's 2 of 20 showed. Check the pilot plan for all three before the go decision. Noah's go decision lists each one.
Exercises
- EX027 — Proposal — List eight proposal headings. Rubric: 3 = eight apt; 2 = six; 1 = three; 0 = none.
- EX028 — Template — Write the tagged template fields. Rubric: 3 = narrative fields plus ATT&CK IDs, no exploit steps; 2 = missing one; 1 = includes exploit steps; 0 = none.
- EX029 — Scope lab — Name two DENY cases assessors must respect. Rubric: 3 = two correct lab reasons; 2 = one; 1 = vague; 0 = wrong.
- EX030 — Tag audit — What does 2/20 wrong tags imply? Rubric: 3 = measurement error, add review, report; 2 = partial; 1 = ignore; 0 = none.
- EX031 — Go criteria — Write four go criteria. Rubric: 3 = four; 2 = three; 1 = one; 0 = none.
- EX032 — UI risk — Ticket UI hides tags — response? Rubric: 3 = no-go until visible; 2 = partial; 1 = continue; 0 = none.
Synthesis
Proposal and pilot turn a question into locked instruments and a realistic go / no-go gate.
Continue studying — chapter 05 of 08
← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →