Chapter 18 — Research methods I
Lesson map
Lesson 18 published. Map, example, EX052.
flowchart TD
N0["Learning outcomes"]
N1["18.1 Questions"]
N2["18.2 Design"]
N3["18.3 Pre-registration"]
N4["18.4 Ethics"]
N5["18.5 Negative results"]
N6["18.7 Writing up a negative result"]
N7["18.8 An internal trial board"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX052"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["18.1 Questions"]
N2["18.2 Design"]
N3["18.3 Pre-registration"]
N4["18.4 Ethics"]
N5["18.5 Negative results"]
N6["18.7 Writing up a negative result"]
N7["18.8 An internal trial board"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX052"]
[Learning outcomes]
|
v
[18.1 Questions]
|
v
[18.2 Design]
|
v
[18.3 Pre-registration]
|
v
[18.4 Ethics]
|
v
[18.5 Negative results]
|
v
[18.7 Writing up a negative result]
|
v
[18.8 An internal trial board]
|
v
[Worked example] --> [Exercise EX052]
Learning outcomes
You will (1) pose answerable questions; (2) design small purple-team trials; (3) pre-register metrics.
18.1 Questions
"Does control X raise detection of T1566 in staging?" beats "is security good?".
A research question has to name what will be measured, under which conditions and against what comparison. Asking whether security is good invites opinions that cannot be wrong and cannot guide a decision. Asking whether a new email filter raises detection of phishing technique T1566 in the staging environment, compared with the current filter, names the control, the technique, the place and the baseline. That question decides the design: a fixed set of phishing samples, a staging mailbox under each filter, and a recorded detection or miss for each sample. Vague questions produce vague work.
18.2 Design
Fixed test set, recorded environment, sample size plan.
Three choices decide whether a later reader can trust the result. The test set is written down before any trial runs, so nobody can drop the samples that make a control look bad. The environment is recorded: which filter version, which mailbox configuration, which date. Without that record, a later change can make the result unreproducible. The sample size is planned for the precision the decision needs, using the same interval methods as detection reporting. A design that skips any of these can still produce a number, but that number will not settle an argument.
18.3 Pre-registration
Write metric and threshold before seeing results.
People naturally adjust what counts as success once they have seen the data. A detection rate of sixty per cent can be called a success or a failure depending on what the team hoped for. Pre-registration writes the metric and the threshold into a dated note before the first trial runs, for example that the new filter succeeds if its Wilson interval's lower bound exceeds the current filter's observed rate. Changing the threshold afterwards is allowed, but it must be labelled as exploratory. The dated note is what stops a hopeful reinterpretation from being presented as the original plan.
18.4 Ethics
Customer data off-limits; synthetic phishing to staff needs HR/CISO approval.
Research that uses real customer messages or accounts is outside this book's methods and TRB's testing agreements. Synthetic phishing sent to real staff is also not an ordinary technical test. It involves people who may feel deceived, and it can affect trust in security messages. Any such trial therefore needs HR and CISO approval that states the purpose, the population, the messaging, the opt-out where feasible and how results will be reported without naming individuals. Technical curiosity is not an approval.
18.5 Negative results
Publish internally; they prevent repeated failed ideas.
A trial that shows a control made no measurable difference is as useful as one that shows it did, because it stops the next team from spending another quarter on the same idea. Negative results also calibrate expectations. If three awareness campaigns have each failed to move click rates in staging exercises, the next proposal for a fourth campaign faces a higher bar. Internal publication means a short write-up on the trial board with the pre-registered metric, the result and a sentence on what not to try again. Hiding negative results makes the organisation relearn the same lesson.
18.6 Limits
Staging ≠ production; state threats to validity.
Staging is quieter, cleaner and differently monitored than production. A detection rate measured there can be higher because traffic is lighter, or lower because production has extra sensors. Threats to validity are the reasons a result might not transfer: the sample was too small, the phishing kit was outdated, the filter was misconfigured, the SOC was told a trial was running. Naming them does not weaken the report. It tells a decision-maker how far to lean on the finding. A result with no stated limits is the one to distrust.
18.7 Writing up a negative result
Negative results get the same template as positive ones. Sam's "MFA prompt fatigue did not improve T1566 detection in n=40" note stopped a second identical trial the following quarter.
The template forces the same fields: question, design, pre-registered threshold, result, limits. Using a different, apologetic format for negative results makes them look like failures of the team rather than findings about a control. The fatigue trial showed that adding more MFA prompts did not raise detection of the phishing set, which redirected the next proposal toward phishing-resistant authenticators instead. Without the write-up, the same fatigue idea would have returned with a new sponsor.
18.8 An internal trial board
A public (internal) board shows pre-registered trials and their status. Duplicate trials are rejected at a glance.
The board lists the question, the owner, the planned sample size and whether the trial is planned, running or written up. Anyone proposing a new trial searches it first. Duplicates are common when teams work in parallel and each believes its idea is original. Rejecting a duplicate early saves a cohort of staff or a block of staging time, and often leads the proposer to join the existing trial instead. The board is intentionally boring. Its job is to make repetition visible.
18.9 Preferring intervals to false significance
Sam avoids claiming significance without a plan. Most purple trials at TRB are descriptive with intervals, not pseudo-clinical trials.
A p-value computed after the fact from a small convenience sample does not mean what people think it means. It can look decisive when the true uncertainty is large. TRB's default is therefore to report counts and Wilson intervals, and to reserve formal significance claims for the rare trials that were designed for them with a sample-size plan. Descriptive reporting is not a lesser form of evidence. It is honest about what a forty-trial purple exercise can support.
18.10 A failed awareness trial still earns a write-up
A pre-registered trial on awareness mail fails its threshold. Sam still writes it up. The next quarter's budget meeting cites the negative result to avoid buying the same content pack again.
The content pack's vendor had promising case studies from other industries. The TRB trial, with a fixed phishing set and a pre-registered detection threshold, did not meet that threshold. Writing it up prevented both a quiet burial and a heated argument based on anecdotes. Budget meetings prefer a dated result to a feeling that the training "seemed useful". The vendor was invited to propose a different pack for a new trial rather than resell the same one.
18.11 Joining an existing trial instead of duplicating it
Registration board rejected a duplicate trial title within a minute. The author joined the existing trial as a helper instead of wasting a cohort.
The duplicate had slightly different wording but the same control, the same technique and the same staging population. Running both would have split the available volunteers and left each trial underpowered. Joining as a helper meant the second author contributed samples and review time, earned co-ownership of the write-up and still answered their underlying question. The board's value is not rejection for its own sake. It is redirecting effort toward a result that can decide something.
18.12 Celebrating negative results
Negative results get a champagne cork icon in the board — celebration of learning, not failure.
Culture decides whether people file negative results or hide them. A small visible reward, even a silly icon, signals that the organisation wants the learning. The icon appears only when the write-up is complete, so it cannot be earned by failing to finish. Teams that publish negative results are thanked in the quarterly research summary alongside teams that publish positive ones. Over a year the board's mix of corks and ordinary ticks shows whether the culture is working.
18.13 Icons do not replace write-ups
Champagne cork icons are optional; the write-up is not. Sam rejects icons without prose.
An icon without a linked write-up would let a team claim credit for a negative result that nobody can learn from. The board's rule is that the icon is rendered only when the write-up fields are filled: question, design, threshold, result, limits. Rejecting a bare icon is occasional and slightly awkward, and it keeps the board useful. Celebration that does not leave a reusable record is just decoration.
18.14 A linter for empty celebrations
Icons without prose are stripped by the board's markdown linter.
The linter is a short check that runs when a board entry is saved. If it finds a cork icon and empty write-up fields, it removes the icon and leaves a comment asking for the prose. Automation matters here because Sam cannot review every edit. The linter also stops well-meaning colleagues from adding icons to old entries that never had write-ups, which would invent a history the board does not have.
Worked example (synthetic TRB)
Sam pre-registers: success if detection ≥80% on 50 T1566-style synthetic emails with Wilson lower bound ≥0.7.
Synthetic case — Riverstone Transit Bank (TRB). The bank, Sam Ito, TrackScope/FindingsDesk/PulseMap and all figures are invented for teaching.
Errors and limits
This book grants no authority to test real banks or payment systems. Labs use documentation-range addresses and synthetic data only; exploit code is never provided.
Exercises
- EX052 — Bad question. Why is "is security good?" a poor research question? Rubric: 3 = not measurable/specific; 2 = partial; 1 = fine; 0 = none.
- EX053 — Pre-register. What should be written before results? Rubric: 3 = metric and threshold (and design); 2 = partial; 1 = vague; 0 = none.
- EX054 — Staff phishing. What approvals before simulated phishing? Rubric: 3 = CISO/HR (and ROE); 2 = one; 1 = vague; 0 = none.
Synthesis
Answerable questions, pre-registered metrics and ethics before purple-team trials.
Continue studying — chapter 18 of 40
← Chapter 17 (draft) · Course chapter index · Chapter 19 (draft) →