Chapter 3 — Study design, ethics and reproducibility
Lesson map
Lesson 03 published. Map, example, EX015.
flowchart TD
N0["Learning outcomes"]
N1["3.1 Design options"]
N2["3.2 Ethics and authorization"]
N3["3.3 Reproducibility plan"]
N4["3.4 Sample size sketch"]
N5["3.5 Stop conditions for source assessments"]
N6["3.5b Data minimisation"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX015"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["3.1 Design options"]
N2["3.2 Ethics and authorization"]
N3["3.3 Reproducibility plan"]
N4["3.4 Sample size sketch"]
N5["3.5 Stop conditions for source assessments"]
N6["3.5b Data minimisation"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX015"]
[Learning outcomes]
|
v
[3.1 Design options]
|
v
[3.2 Ethics and authorization]
|
v
[3.3 Reproducibility plan]
|
v
[3.4 Sample size sketch]
|
v
[3.5 Stop conditions for source assessments]
|
v
[3.5b Data minimisation]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX015]
Learning outcomes
You will (1) choose a study design that compares reporting formats fairly; (2) define inclusion and ethics rules; (3) plan reproducibility from day one; (4) size the sample from a pilot guess; and (5) write stop conditions for the assessment work that generates findings.
3.1 Design options
A between-findings design randomly assigns each synthetic finding to ATT&CK-tagged or narrative-only report text, holding severity and asset class fixed. That yields many units and supports stratification.
A between-engagement design alternates report formats by week. It is simpler operationally but confounds format with time and team load. Prefer between-findings when the ticket system can store a format flag.
For Noah, the format flag can be stored in CBM's ticket system, so between-findings randomisation is possible. Holding severity and asset class fixed matters because tagged findings could otherwise cluster in high-severity tickets that are fixed faster for that reason alone.
3.2 Ethics and authorization
All assessments that produce study findings run under written authorization on synthetic segments only. Operators and remediators consent to research use of ticket metadata. No real customer data enters the dissertation.
Dual-use rule: findings describe impact and evidence references, never exploit steps. If a training assessment unexpectedly touches a production host, stop conditions fire and research pauses with the assessment.
Noah files the authorization and remediator consent before the first finding enters the study. Consent forms state the purpose, the ticket metadata used, the right to withdraw and a supervisor contact.
3.3 Reproducibility plan
Publish the randomisation script, both report templates, the operational definition of remediated, and a synthetic ticket export schema. Pin the ATT&CK version used for tags. A peer should regenerate your tables from the synthetic export.
Each item answers a question an examiner may ask. The script shows how findings were assigned; the templates show what remediators saw; the definition shows what counted as remediated; and the schema lets others check the export. With the ATT&CK version pinned, tags written at the start of the study keep the same meaning at the end, even after framework updates.
3.4 Sample size sketch
If narrative completion is about 40% in 30 days, detecting a rise to 60% needs on the order of 100 findings per arm for a rough comparison. Treat this as a planning sketch; refine after the pilot's variance estimate.
The sketch also warns about feasibility. If CBM's training assessments produce only a few dozen findings a month, reaching 100 per arm may take a full term or more. Noah records this as a timeline risk in his proposal. The pilot in Chapter 5 is meant to improve the estimate, not to answer the question.
3.5 Stop conditions for source assessments
Out-of-scope host, appearance of real personal data, or availability impact on shared services — stop, escalate, and pause research data collection until cleared. Research never pressures assessors to continue unsafely.
"""Evaluate proposed actions against synthetic rules of engagement."""
ROE = {
"hours": ("09:00", "17:00"),
"deny_actions": {"denial-of-service", "phishing-staff", "exfiltrate-real-data"},
"require_approval": {"password-spray-lab", "privilege-escalation-lab"},
"max_concurrency": 5,
}
proposals = [
("service-enum", "10:30", 3, False),
("denial-of-service", "10:30", 1, False),
("password-spray-lab", "10:30", 2, False),
("password-spray-lab", "10:30", 2, True),
("service-enum", "18:00", 2, False),
("service-enum", "11:00", 20, False),
]
def ok_time(t):
return ROE["hours"][0] <= t <= ROE["hours"][1]
for action, when, conc, approved in proposals:
reasons=[]
if action in ROE["deny_actions"]: reasons.append("deny-action")
if action in ROE["require_approval"] and not approved: reasons.append("needs-approval")
if not ok_time(when): reasons.append("outside-hours")
if conc > ROE["max_concurrency"]: reasons.append("concurrency")
print(f"action={action:20} when={when} conc={conc:2} approved={approved} -> {'ALLOW' if not reasons else 'DENY'} {','.join(reasons) or '-'}")
Expected output:
action=service-enum when=10:30 conc= 3 approved=False -> ALLOW -
action=denial-of-service when=10:30 conc= 1 approved=False -> DENY deny-action
action=password-spray-lab when=10:30 conc= 2 approved=False -> DENY needs-approval
action=password-spray-lab when=10:30 conc= 2 approved=True -> ALLOW -
action=service-enum when=18:00 conc= 2 approved=False -> DENY outside-hours
action=service-enum when=11:00 conc=20 approved=False -> DENY concurrency
The ROE lab shows the same logic for proposed actions: service enumeration inside working hours is allowed; denial of service is denied; a lab password spray is denied until approved; work at 18:00 or with concurrency 20 is denied. Findings produced by work outside these rules never enter the study data.
3.5b Data minimisation
Ticket exports for research should drop names where possible, keep role IDs, and exclude free-text that might hold credentials. Store exports encrypted at rest in the university drive, with access limited to Noah and supervisors. Retention ends at dissertation archive freeze plus one year unless ethics requires sooner deletion.
Noah's export drops remediator names and free-text fields. Role IDs keep the analysis possible without naming people: Noah can still check that writers did not grade their own findings. Free text is excluded because remediators sometimes paste configuration snippets or passwords into ticket comments. The retention rule is written into the ethics application and the data management plan, so the deletion date is checked rather than remembered.
3.6 Worked example (synthetic CBM case)
Noah chooses between-findings randomisation on CBM synthetic VLAN 40. Authorization and remediator consent are filed. The ROE lab pattern — deny unsafe actions, require approval for sensitive labs, enforce hours — is mirrored in assessment runbooks so study data only accrues from well-governed work.
VLAN 40 is a synthetic training segment, so no customer data is involved. The runbooks mean that a denied or unapproved action is stopped before it can produce a finding, and research data comes only from work the authorization allows. The design, authorization and consent are filed with the supervisor before collection starts.
3.7 Errors and limits
- Randomising after seeing outcomes.
- Using live customer segments.
- No definition of remediated.
Each error is costly to fix later. Randomising after seeing outcomes lets the researcher steer findings into the arm that suits the hypothesis, which destroys the comparison. Live customer segments expose real data and break the authorization. Without a definition of remediated, each ticket is judged differently, and the primary measure has no meaning. Check the design for all three before collection begins. Noah's design addresses each one in writing.
Exercises
- EX015 — Design — Compare between-findings vs between-engagement designs. Rubric: 3 = trade-offs on power and confounders; 2 = one design; 1 = vague; 0 = none.
- EX016 — Ethics — List five ethics controls. Rubric: 3 = five apt; 2 = four; 1 = two; 0 = none.
- EX017 — Remediated — Define remediated operationally. Rubric: 3 = verifiable criteria and who checks; 2 = partial; 1 = vague; 0 = none.
- EX018 — ROE link — Why research inherits assessment stop conditions. Rubric: 3 = safety first, data quality; 2 = one; 1 = vague; 0 = none.
- EX019 — Sample — Critique the 100-per-arm sketch. Rubric: 3 = assumptions, pilot need, feasibility; 2 = two; 1 = one; 0 = none.
- EX020 — Consent — Write remediator consent bullets. Rubric: 3 = purpose, data, withdrawal, contacts; 2 = three; 1 = one; 0 = none.
Synthesis
Design, ethics and reproducibility decisions are made before tickets accrue — they are not afterthoughts.
Continue studying — chapter 03 of 08
← Chapter 02 (draft) · Course chapter index · Chapter 04 (draft) →