◇ OpenCyber Academy
Technologist · Chapter 24 · English

Chapter 24 — Applied research methods

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 24 — Applied research methods

Lesson map

Lesson 24 published. Map, example, EX070.

flowchart TD
  N0["Learning outcomes"]
  N1["24.1 Questions"]
  N2["24.2 Design"]
  N3["24.3 Pitfalls"]
  N4["24.4 Write-up"]
  N5["24.4b Ethics in applied studies"]
  N6["24.7 Sharing results inside the company"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX070"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["24.1 Questions"]
  N2["24.2 Design"]
  N3["24.3 Pitfalls"]
  N4["24.4 Write-up"]
  N5["24.4b Ethics in applied studies"]
  N6["24.7 Sharing results inside the company"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX070"]

[Learning outcomes]
    |
    v
[24.1 Questions]
    |
    v
[24.2 Design]
    |
    v
[24.3 Pitfalls]
    |
    v
[24.4 Write-up]
    |
    v
[24.4b Ethics in applied studies]
    |
    v
[24.7 Sharing results inside the company]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX070]

Learning outcomes

You will (1) turn a practical question into a testable one; (2) design a small comparative study; (3) avoid common pitfalls in security ML studies; and (4) write up findings for colleagues.

24.1 Questions

Practical questions become testable when they name the intervention, the comparison, the measure and the setting: "Does separating untrusted reviews into a different prompt section reduce harmful tool calls compared with the current prompt, on Ask Brightwell staging, measured on 200 held-out injection cases?" A good question also says what result would change a decision. Jonas's question for Ask Brightwell named all four, and also the decision: if the separated prompt reduced harmful proposals, it would be adopted as an extra layer.

24.2 Design

Compare conditions on the same cases; randomise order; repeat with several seeds; fix versions. Decide sample size before running: to detect a drop from 20% to 10% you need around 200 cases per condition. Write the analysis plan in advance. For the separated-prompt study, Jonas ran both prompts on the same 200 held-out cases with five seeds each, with the vendor model version pinned. The sample size was chosen from the planned drop before running, and the analysis plan was written first, so the comparison could not be redesigned after seeing the numbers.

24.3 Pitfalls

Arp et al. (arXiv:2010.09470) list pitfalls in ML for security, including sampling bias (test cases unlike real attacks), data snooping (tuning on the test set), inappropriate baselines and lab-only evaluation. Guard against each explicitly in the design. Jonas's study names its main pitfall: the cases were written by his team, which may not match real attacks. He avoided data snooping by using held-out cases and compared against the current prompt as the baseline, which is the decision actually facing Brightwell.

24.4 Write-up

One to three pages: question, method, results table with intervals, interpretation, limits, recommendation. Attach scripts and case lists so colleagues can rerun the study.

"""Lab S: reporting attack-success rates honestly: Wilson intervals,
seed variation and a paired bootstrap. Synthetic trial outcomes only."""
import math, random
def wilson(k, n, z=1.96):
    p = k/n; d = 1 + z*z/n
    c = (p + z*z/(2*n))/d; h = z*math.sqrt(p*(1-p)/n + z*z/(4*n*n))/d
    return c - h, c + h
for k, n in ((3, 20), (30, 200), (300, 2000)):
    lo, hi = wilson(k, n); print(f"successes={k}/{n} rate={k/n:.3f} wilson95=[{lo:.3f}, {hi:.3f}]")
# five seeds of a synthetic defence evaluation: per-seed attack success rate
rates = []
for seed in range(5):
    r = random.Random(seed); rates.append(sum(r.random() < 0.12 for _ in range(200))/200)
m = sum(rates)/5; sd = math.sqrt(sum((x-m)**2 for x in rates)/4)
print("per-seed rates:", " ".join(f"{x:.3f}" for x in rates), f"mean={m:.3f} sd={sd:.3f}")
# paired bootstrap: baseline vs defence on the same 300 synthetic prompts
r = random.Random(42)
base = [r.random() < 0.30 for _ in range(300)]
defn = [b and r.random() < 0.6 for b in base]   # defence blocks ~40% of successes
diffs = []
for _ in range(2000):
    idx = [r.randrange(300) for _ in range(300)]
    diffs.append(sum(base[i] for i in idx)/300 - sum(defn[i] for i in idx)/300)
diffs.sort()
print(f"baseline_asr={sum(base)/300:.3f} defended_asr={sum(defn)/300:.3f} "
      f"reduction_ci95=[{diffs[50]:.3f}, {diffs[1949]:.3f}]")

Expected output:

successes=3/20 rate=0.150 wilson95=[0.052, 0.360]
successes=30/200 rate=0.150 wilson95=[0.107, 0.206]
successes=300/2000 rate=0.150 wilson95=[0.135, 0.166]
per-seed rates: 0.110 0.130 0.135 0.105 0.145 mean=0.125 sd=0.017
baseline_asr=0.350 defended_asr=0.190 reduction_ci95=[0.120, 0.203]

Jonas's write-up of the separated-prompt study fits on two pages and ends with the recommendation to adopt it as an extra layer while keeping the policy service. The case list and scripts sit in the repository next to the note. The study's scripts run on staging and print the same rates from the stored seeds, so a colleague can confirm the 21% and 12% figures without asking Jonas for anything.

24.4b Ethics in applied studies

Studies use synthetic data and staging systems under written authorisation. If a study uncovers a live vulnerability, stop and report it through the incident process before continuing. For Jonas's study, all cases ran on Ask Brightwell staging with test accounts under his written authorisation. Had a case revealed a weakness that also affected production, the study would have paused while it went through the incident process. Written authorisation names the staging environment and test accounts, and any change of scope goes back to the same approver.

24.5 Worked example (synthetic Brightwell case)

Jonas compares the current prompt with a "separated sections" prompt on 200 held-out cases, five seeds each. Harmful proposals: 21% versus 12%. Like the lab's comparison (35% to 19%, reduction interval 12–20 points), he reports the difference with an interval. Limitation: cases written by the team. Recommendation: adopt the separated prompt as an extra layer; keep the policy service. The result supports the separated prompt as an added layer, not as a replacement for enforcement. The interval tells owners how large the reduction plausibly is, and the limitation tells them why the study cannot promise the same effect against real attackers.

24.6 Errors and limits

Jonas's study avoids these: the question names prompt, measure and setting; the sample size was set before the run; and only held-out cases were used. Reusing tuning cases would have flattered the separated prompt, and a vague question would have left nothing to decide. The five seeds and the pinned vendor model version were fixed at the same time. Each choice is in the study plan, dated before the run, so anyone can check that nothing was changed after seeing the results.

24.7 Sharing results inside the company

A short internal note summarises each applied study for the product and support teams: the question, what was found, how confident the team is and what decision it supports. Detailed methods stay in the appendix for anyone who wants to check them. For the separated-prompt study, the note told product and support that the new prompt reduced harmful proposals on the team's cases, that the policy service still decides, and that the cases may not match real attacks. The note links to the study's scripts and case list for anyone checking it.

Exercises

Synthesis

Applied research in practice means testable questions, pre-planned comparisons, guarded pitfalls and short, reproducible write-ups.

Continue studying — chapter 24 of 30

← Chapter 23 (draft) · Course chapter index · Chapter 25 (draft) →