Chapter 3 — Original contribution proposal | Doctoral research Authorized offensive security and red team

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 03 published. Map, example, EX015.

flowchart TD
  N0["Learning outcomes"]
  N1["3.1 Falsifiable questions"]
  N2["3.2 Pre-registration"]
  N3["3.3 Ethical protocol"]
  N4["3.4 Baselines and sample size"]
  N5["3.5 Approval"]
  N6["3.5b Failure modes of the proposal"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX015"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["3.1 Falsifiable questions"]
  N2["3.2 Pre-registration"]
  N3["3.3 Ethical protocol"]
  N4["3.4 Baselines and sample size"]
  N5["3.5 Approval"]
  N6["3.5b Failure modes of the proposal"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX015"]

[Learning outcomes]
    |
    v
[3.1 Falsifiable questions]
    |
    v
[3.2 Pre-registration]
    |
    v
[3.3 Ethical protocol]
    |
    v
[3.4 Baselines and sample size]
    |
    v
[3.5 Approval]
    |
    v
[3.5b Failure modes of the proposal]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX015]

Learning outcomes

You will (1) write a falsifiable research question; (2) specify hypotheses, measures and analysis before collecting data; (3) design an ethical protocol for authorized lab studies; (4) plan baselines and sample sizes; and (5) obtain supervisory and ethics approval.

3.1 Falsifiable questions

A good question can be wrong. "Do machine-checkable scope predicates reduce out-of-scope probes compared with prose ROE, in NHUSTL's authorized labs, measured as probes per engagement-hour?" can fail. "How can red teams be better?" cannot.

Elena's question names the population (NHUSTL authorized labs), the intervention (predicate checker on or off), and the measure (out-of-scope probes per engagement-hour). A result of no difference would still be publishable if the study was powered and executed as planned. Write the question so a null result is still a clear answer.

3.2 Pre-registration

Write down: population (engagements on the lab range), intervention (predicate checker on/off), primary measure (out-of-scope probes / hour), secondary measures (time-to-stop, false stops), analysis (rate difference with interval), exclusions (aborted for infra failure). File it before the main study.

Elena files hers before the first main-study engagement. The file also records the ATT&CK and checker versions, so later readers know which definitions applied. A peer should be able to run the main study from the file alone. File it where the supervisor and ethics board can both find it.

3.3 Ethical protocol

Participants (student operators) consent; targets are synthetic; no real credentials; dual-use appendix explains what will not be published. Align with institutional ethics and with NIST SP 800-115's emphasis on careful planning.

"""Evaluate proposed actions against synthetic rules of engagement."""
ROE = {
    "hours": ("09:00", "17:00"),
    "deny_actions": {"denial-of-service", "phishing-staff", "exfiltrate-real-data"},
    "require_approval": {"password-spray-lab", "privilege-escalation-lab"},
    "max_concurrency": 5,
}
proposals = [
    ("service-enum", "10:30", 3, False),
    ("denial-of-service", "10:30", 1, False),
    ("password-spray-lab", "10:30", 2, False),
    ("password-spray-lab", "10:30", 2, True),
    ("service-enum", "18:00", 2, False),
    ("service-enum", "11:00", 20, False),
]

def ok_time(t):
    return ROE["hours"][0] <= t <= ROE["hours"][1]

for action, when, conc, approved in proposals:
    reasons=[]
    if action in ROE["deny_actions"]: reasons.append("deny-action")
    if action in ROE["require_approval"] and not approved: reasons.append("needs-approval")
    if not ok_time(when): reasons.append("outside-hours")
    if conc > ROE["max_concurrency"]: reasons.append("concurrency")
    print(f"action={action:20} when={when} conc={conc:2} approved={approved} -> {'ALLOW' if not reasons else 'DENY'} {','.join(reasons) or '-'}")

Expected output:

action=service-enum         when=10:30 conc= 3 approved=False -> ALLOW -
action=denial-of-service    when=10:30 conc= 1 approved=False -> DENY deny-action
action=password-spray-lab   when=10:30 conc= 2 approved=False -> DENY needs-approval
action=password-spray-lab   when=10:30 conc= 2 approved=True -> ALLOW -
action=service-enum         when=18:00 conc= 2 approved=False -> DENY outside-hours
action=service-enum         when=11:00 conc=20 approved=False -> DENY concurrency

The ROE lab shows the same discipline for proposed actions: denial of service is denied; a lab password spray needs approval; work at 18:00 or with concurrency 20 is denied. Findings from work outside these rules never enter study data. Consent forms state purpose, data used, withdrawal rights and contacts.

3.4 Baselines and sample size

Estimate the prose-ROE out-of-scope rate from a pilot. Choose a sample (engagements or hours) that can detect a meaningful reduction — for example, from 0.20 to 0.05 probes/hour — with a planned interval width.

Elena's pilot suggests a prose rate near 0.18 per hour. Detecting a drop towards 0.05 with a usable interval needs on the order of 20 two-hour engagements per arm, which is feasible across two terms at NHUSTL. Record the sketch as a planning figure, and refine it after the pilot's variance estimate.

3.5 Approval

Supervisor sign-off on science; ethics board on people; lab owner on systems. Keep versions of the proposal with dates.

Elena files all three before the pilot. A change to the proposal after approval needs a dated amendment, not an email. Without all three, the pilot does not start. Keep the signed copies with the pre-registration, and check that the dates still cover the work before each engagement. An expired lab authorization or ethics approval pauses the research as well as the assessment.

3.5b Failure modes of the proposal

Common failure modes: a question that only the intervention arm can "win" by construction; measures that punish reporting (operators hide probes); ethics that block independent validation. Stress-test the proposal by asking what result would make you abandon the idea, and whether that result is observable.

For Elena, a result that would abandon the idea is a checker that does not reduce out-of-scope probes once false stops are fixed. That result is observable from the primary measure. A measure that punished operators for reporting probes would hide the very events the study needs.

3.6 Worked example (synthetic NHUSTL case)

Elena's proposal: H1 — predicate checking halves out-of-scope probe rate vs prose ROE. Primary measure: denied or executed probes outside CIDR/port/action lists per engagement-hour. The ROE lab shows deny-action, outside-hours and concurrency rules blocking unsafe proposals before they run. Main-study target: 20 engagements per arm, after a pilot of 10 per arm. Ethics: student operators, synthetic targets, no publication of working attack procedures.

H1 can fail: if the checker rate matches the prose rate within the planned interval, the hypothesis is not supported. The ethics paragraph is short enough for the supervisor to quote when asked what will not be published.

3.7 Errors and limits

Hypotheses written after looking at data.
Pilots that use real client systems.
Proposals without a stop-condition plan.

Each error damages the proposal. Hypotheses written after looking at data cannot be falsified honestly. Pilots on real client systems break authorization and expose real data. Proposals without stop conditions leave operators without a rehearsed pause. Check the proposal for all three before seeking approval. Elena's proposal addresses each one in writing before she seeks signatures, so examiners can see the controls were planned.

Exercises

EX015 — Question. Rewrite a vague goal as a falsifiable question. Rubric: 3 = population, intervention, measure; 2 = two; 1 = one; 0 = none.
EX016 — Pre-reg. List eight pre-registration fields. Rubric: 3 = eight apt; 2 = six; 1 = three; 0 = none.
EX017 — ROE lab. Explain two DENY rows. Rubric: 3 = both with rules; 2 = one; 1 = vague; 0 = wrong.
EX018 — Ethics. List four ethics controls for Elena's study. Rubric: 3 = four; 2 = three; 1 = one; 0 = none.
EX019 — Sample. Argue for 20 engagements per arm. Rubric: 3 = effect size, variance, feasibility; 2 = two; 1 = assertion; 0 = none.
EX020 — Approvals. Who must sign what? Rubric: 3 = supervisor, ethics, lab owner with artefacts; 2 = two; 1 = one; 0 = none.

Synthesis

A doctorate proposal commits to a falsifiable question, pre-registered measures, ethics that bound harm and approvals before data collection.

Continue studying — chapter 03 of 08

← Chapter 02 (draft) · Course chapter index · Chapter 04 (draft) →