Chapter 42 — Authorized offensive security and red team: applied research
Lesson map
Lesson 42 published. Map, example, EX124.
flowchart TD
N0["Learning outcomes"]
N1["42.1 Questions that fit"]
N2["42.2 Design"]
N3["42.3 Disclosure"]
N4["42.2b Pre-registration"]
N5["42.3b Negative results publishability"]
N6["42.1b Data sharing agreements"]
N7["Practice note"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX124"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["42.1 Questions that fit"]
N2["42.2 Design"]
N3["42.3 Disclosure"]
N4["42.2b Pre-registration"]
N5["42.3b Negative results publishability"]
N6["42.1b Data sharing agreements"]
N7["Practice note"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX124"]
[Learning outcomes]
|
v
[42.1 Questions that fit]
|
v
[42.2 Design]
|
v
[42.3 Disclosure]
|
v
[42.2b Pre-registration]
|
v
[42.3b Negative results publishability]
|
v
[42.1b Data sharing agreements]
|
v
[Practice note]
|
v
[Worked example] --> [Exercise EX124]
Learning outcomes
You will (1) frame a research question about assessment platforms; (2) choose measures and baselines; (3) keep dual-use out of student research.
42.1 Questions that fit
"Does dual-approval reduce scope-widen incidents?" "Does showing denominators change stakeholder interpretation of coverage?" Not "new bypass against product X".
Questions about dual approval and denominator presentation improve the platform; questions about bypasses against a vendor product do not belong in this degree. Sam returns proposals that need live exploitation and asks for a measurement question instead. The two example questions in the stem are the template for what advances. Measurement questions can still be ambitious; they just stay inside authorization.
42.2 Design
Pre-register measures; use synthetic or consented operational data; report intervals. Replication beats novelty theatre.
Pre-registration and intervals are how applied work stays distinguishable from dashboard storytelling. Synthetic or consented data keeps ethics intact. Sam files the measures in the repo before any outcome CSV is opened, and the commit timestamp is part of the thesis appendix. Opening the CSV first is treated as a process defect in studio grading. The pre-registration commit is linked from the thesis methods chapter so examiners can see the date.
42.3 Disclosure
If research accidentally finds a live issue, follow responsible disclosure — not a class blog.
Synthetic case — Riverstone Transit. Sam's applied study compared ticket reopen rates before/after requiring re-test evidence; pre-registered, synthetic+anonymized.
A class blog is not a disclosure channel. Live issues found while studying reopen rates follow the host's responsible path and stay out of coursework until cleared. Sam's reopen-rate study used synthetic and anonymized tickets only; anything that looked like a live credential path was escalated to Ingrid, not written up for class.
42.4 Errors and limits
Course projects are not IRB-proofed human-subjects work; avoid collecting personal data.
Collecting rider or staff identifiers for a "richer" thesis is out of bounds even with good intentions. Stick to synthetic and agreed aggregates. Sam's ethics paragraph states that limit explicitly so examiners do not have to guess. Aggregates that can re-identify a single staff member are refused the same way raw dumps are. Coursework that needs IRB-style consent is redesigned until it no longer needs it. Sam's ethics paragraph names that redesign duty explicitly.
42.2b Pre-registration
Write the hypothesis, primary measure and exclusion rules before looking at outcome data. File them in the repo. Changing the measure after seeing results is how empty scores return through the research door.
Filing hypothesis, measure and exclusions first makes that move visible as a deviation. Sam's dual-approval study kept reopen rate as primary even when a secondary metric looked more flattering. Deviations require a dated note in the appendix, not a quiet edit to the notebook. The filing commit is dated before any outcome data is pulled, which reviewers can check in the history.
42.3b Negative results publishability
Publishable applied research includes negative results ("dual approval did not change reopen rates in this sample"). Hiding negatives biases the next cohort's designs toward empty confidence.
A negative result on dual approval still teaches the next cohort not to assume the control works without evidence. Sam's write-up leads with the negative and discusses sample size before suggesting a follow-up. Cohorts that only see positive write-ups keep shipping unmeasured controls. Studio prompts now include a negative-result example from Sam's write-up. Negative results are reported with the same intervals as positive ones.
42.1b Data sharing agreements
Even synthetic-plus-anonymized operational metrics need a written agreement with the host for student theses. "My mentor said it was fine" is not a sharing agreement.
Verbal blessing from a mentor does not satisfy residency or thesis rules. A written agreement names what may leave the host and in what form. Sam's placement host signed a one-page metrics agreement before any anonymized counts went into a draft thesis. Without that page, the thesis chapter on results stays empty rather than informal.
Practice note
Preregistered analysis code lives in the repo with a freeze commit before data collection. Amending the analyzer after peeking at outcomes requires a logged amendment with reason. This is lightweight compared with clinical trials and still blocks the most common form of accidental p-hacking in undergraduate studies.
State the dual-use boundary in the thesis ethics section with a concrete refusal example.
Negative results get the same figure quality as positive ones in the thesis template.
42.4b Ethics review for platform research
A study comparing detection coverage across engagements uses client data, even if summarised. Sam's proposal goes through the organisation's ethics and legal review, uses only aggregated, anonymised counts, and has each client's written consent before any result is shared outside the team.
Aggregated coverage counts are still client data until consent and review say otherwise. Sam's proposal listed each client, the aggregation rule and the ban on per-host rows outside the team. Sharing outside the team waited on written consent from every client in the sample.
Exercises
- EX124 — Question. Write one acceptable research question. Rubric: 3 = platform/process focused; 2 = borderline; 1 = exploit focused; 0 = none.
- EX125 — Measure. Name primary measure for dual-approval study. Rubric: 3 = e.g. unauthorized widen attempts/month; 2 = vague; 1 = "security"; 0 = none.
- EX126 — Dual-use. Refuse a project that asks for a new exploit. Rubric: 3 = refusal + allowed alternative; 2 = refusal; 1 = accepts; 0 = none.
Synthesis
Research the platform and process; leave weaponization alone.
Continue studying — chapter 42 of 50
← Chapter 41 (draft) · Course chapter index · Chapter 43 (draft) →