Chapter 24 — Applied research methods | Technologist Authorized offensive security and red team

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 24 published. Map, example, EX070.

flowchart TD
  N0["Learning outcomes"]
  N1["24.1 Questions worth asking"]
  N2["24.2 Methods that fit technologists"]
  N3["24.3 Ethics first"]
  N4["24.4 Abstract checklist"]
  N5["24.5 Reproducibility for technologists"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX070"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["24.1 Questions worth asking"]
  N2["24.2 Methods that fit technologists"]
  N3["24.3 Ethics first"]
  N4["24.4 Abstract checklist"]
  N5["24.5 Reproducibility for technologists"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX070"]

[Learning outcomes]
    |
    v
[24.1 Questions worth asking]
    |
    v
[24.2 Methods that fit technologists]
    |
    v
[24.3 Ethics first]
    |
    v
[24.4 Abstract checklist]
    |
    v
[24.5 Reproducibility for technologists]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX070]

Learning outcomes

You will (1) pose a research question about assessment practice that can be answered with synthetic or authorized data; (2) choose a method (measurement study, case comparison, structured interview); (3) state ethics and limits up front; (4) write an abstract checklist that catches missing pieces.

24.1 Questions worth asking

Examples: Does a three-gate pipeline reduce out-of-scope attempts in student exercises? Which ATT&CK techniques does TRT's SOC alert on after one purple cycle? How long do remediation tickets of each severity stay open? Each question names a population, a measure and a limit.

A question such as whether red teaming works fails this test, because it names no population and no measure. Priya's studio question about out-of-scope attempts passes: the population is studio exercises, the measure is OUT-OF-SCOPE lines in the checker log, and the limit is that student behaviour may differ from a contracted team. Questions that could only be answered by testing real users without their knowledge are rejected outright.

24.2 Methods that fit technologists

Measurement studies on ticket and coverage data; before/after comparisons when a control changes; structured interviews with SOC analysts about alert quality. Laboratory exploit research is out of scope for this course.

The common thread is data a technologist already handles: ticket timestamps, coverage matrix cells and the words of analysts who triage the alerts. A before and after comparison around DET-19, for example, can count how often the new detection fired on replayed synthetic activity. Proposals that drift towards novel exploit development are redirected, because they need a laboratory and an authorisation this course cannot provide.

24.3 Ethics first

No covert testing of real users, no collection of personal data beyond the agreement, no publication of client identifiers. Label synthetic data. Obtain approval before interviewing staff.

Each of these rules has a concrete check that a reviewer can run before submission. Synthetic logs carry a label in their file names and headers, interview consent is written and filed before any abstract is drafted, and every public draft is searched for client names and identifiers before it is uploaded to the seminar. A draft that fails the search goes back to its author rather than being edited quietly by a reviewer.

24.4 Abstract checklist

A passable abstract states the question, the method, the sample size n, the limits and the ethics posture. Missing n is a common failure mode in student papers.

Priya's worked example shows the minimum: n = 3 exercises and 12 actions, stated in the abstract next to the limit that the data are synthetic. Seminar grading treats a missing limits line as seriously as a missing n, because both let a reader overestimate what the study shows. The ethics line names the posture, here no live systems, rather than saying ethics was considered.

24.5 Reproducibility for technologists

A classmate should be able to rerun the counts from the synthetic logs and obtain the same fractions. Priya stores the input files beside the abstract and notes the Python version. Reproducibility here is not about exotic environments; it is about not waving at numbers that nobody else can rebuild.

For the HOLD-fraction mini-study this means the three synthetic exercise logs, the counting script and a note of the Python version sit in one folder beside the abstract. A classmate who reruns the script should arrive at the same count of HOLDs out of 12 actions. A screenshot of a spreadsheet total is not accepted, because nobody can check how it was produced.

Worked example

Priya's mini-study asks: after introducing the operator log, what fraction of planned actions were HOLDs in three synthetic exercises? Method: count pipeline outputs. n = 3 exercises, 12 actions. Limit: synthetic only. Ethics: no live systems. She refuses to claim the result generalises to all vendors.

Synthetic case. The mini-study design is invented for coursework.

Errors and limits

Harvesting production logs for a paper without approval is still a data-handling breach. Tiny n needs humble claims.

Exercises

EX070 — Question shape. Rewrite "Are red teams good?" as a measurable research question for TRT.
Show marking criteria
Rubric: 3 = names measure, population, timeframe; 2 = partly measurable; 1 = still vague; 0 = none.
EX071 — Missing n. An abstract describes method and limits but not sample size. What does the checklist say?
Show marking criteria
Rubric: 3 = n MISSING → revise; 2 = incomplete; 1 = fine as-is; 0 = none.
EX072 — Ethics line. Write one ethics sentence Priya can put in her abstract.
Show marking criteria
Rubric: 3 = no live systems / synthetic or authorized data / no personal data beyond agreement; 2 = vague ethics; 1 = none needed; 0 = none.

Synthesis

Ask a measurable question, pick a humble method, put ethics in the abstract, and never skip n.

Continue studying — chapter 24 of 30

← Chapter 23 (draft) · Course chapter index · Chapter 25 (draft) →