◇ OpenCyber Academy
Technologist · Chapter 24 · English

Chapter 24 — Applied research methods

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 24 — Applied research methods

Lesson map

Lesson 24 published. Map, example, EX070.

flowchart TD
  N0["Learning outcomes"]
  N1["24.1 The question"]
  N2["24.2 Data"]
  N3["24.3 Measure"]
  N4["24.4 Comparison"]
  N5["24.5 Ethics"]
  N6["24.7 Next step"]
  N7["Literature"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX070"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["24.1 The question"]
  N2["24.2 Data"]
  N3["24.3 Measure"]
  N4["24.4 Comparison"]
  N5["24.5 Ethics"]
  N6["24.7 Next step"]
  N7["Literature"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX070"]

[Learning outcomes]
    |
    v
[24.1 The question]
    |
    v
[24.2 Data]
    |
    v
[24.3 Measure]
    |
    v
[24.4 Comparison]
    |
    v
[24.5 Ethics]
    |
    v
[24.7 Next step]
    |
    v
[Literature]
    |
    v
[Worked example] --> [Exercise EX070]

Learning outcomes

You will (1) frame a practical research question in a managed SOC; (2) collect and analyse operational data ethically; (3) use simple comparisons and descriptive statistics; (4) present findings with limits.

24.1 The question

Mei asks: "Do clients who received a 15-minute security briefing report suspicious emails more often than those who did not?"

Mei keeps the question narrow on purpose so that it can be answered. It names one intervention, the 15-minute briefing, one outcome, reports of suspicious emails, and one comparison, briefed against unbriefed clients. A broader question such as whether briefings make clients safer would have no single measure that Mei could count from Copperleaf's data. Mei writes the question down before looking at any data, so the comparison cannot be reshaped afterwards to fit an interesting result.

24.2 Data

Copperleaf briefed twelve clients last year as part of onboarding; eighteen others were not briefed. The phishing report mailbox records reports by client.

Thirty clients in total, twelve briefed and eighteen not, is the sample the data allow. The groups were not chosen by Mei: briefing began as part of onboarding, which is why the newer clients are the briefed ones. The phishing report mailbox already tags each report with its client, so no new data collection was needed. Mei also notes that the mailbox only counts emails people chose to report, so it says nothing about suspicious emails nobody noticed.

24.3 Measure

Reports per 100 staff per quarter, so client sizes are comparable.

A client with sixty staff will send more reports than one with ten, even if each person behaves the same. Dividing by staff numbers and fixing the period to one quarter puts a small clinic and a larger firm on the same scale. Mei takes staff numbers from the client profiles, which list them for licensing. A quarter is long enough to smooth out a single busy week, yet short enough to compare with the follow-up Mei proposes for next quarter. The measure is written down before the comparison is run, together with how staff numbers are counted.

24.4 Comparison

Briefed clients averaged 4.1 reports per 100 staff per quarter; others 1.6. The difference is large, but the groups were not randomly chosen: briefed clients joined more recently and may differ in other ways.

The ratio is 4.1 divided by 1.6, about 2.6, which Mei rounds rather than quoting more decimals than the data justify. Because newer clients may have newer email systems or staff more used to reporting, the difference may come partly or wholly from those other features rather than from the briefing itself.

24.5 Ethics

Data are aggregated per client; no individual staff are identified. Clients agreed to anonymised use in Copperleaf's improvement work.

Working with counts per client rather than per person means no analyst sees which receptionist did or did not report an email. That matters at small clients, where a single report could otherwise point to one named person. The agreement with clients covers use in improvement work only, not publication outside Copperleaf. The ethics note goes into Mei's write-up, so readers know how the data were handled before they see any figure.

24.6 Limits

Confounding by joining date and client type; small numbers; reports are not the same as resisting phishing.

Each limit points to a specific weakness in what the comparison can show. Joining date and client type could explain the gap without any effect of the briefing; twelve and eighteen clients are too few for stable averages; and a report shows that someone noticed an email, not that others in the same business did not click it. Mei lists these limits in the summary for Raj, beside the figures rather than in a footnote that a busy reader would skip.

24.7 Next step

Mei proposes briefing six of the unbriefed clients next quarter, chosen by lot, and comparing with the other twelve unbriefed — a fairer test.

Choosing by lot means Mei cannot, even unintentionally, pick the clients most likely to report anyway. The six briefed and twelve unbriefed clients would then differ mainly in whether they received the briefing, so a difference next quarter would be easier to attribute to it, though six clients is still a small group. The proposal keeps the same measure, reports per 100 staff per quarter, so the follow-up result can be compared directly with this one.

Literature

Mei reads Alahmadi et al. (2022) on how analysts experience alarms and notes that user reports are a form of human sensor. She also reads Sundaramurthy et al. (2014), whose researchers worked inside SOCs to understand practice, which encourages her to talk to client contacts about why they report or do not, rather than relying on numbers alone.

Presenting

Mei presents the result to Copperleaf's team with a chart of rates per client group, the caveats and the proposed follow-up, in ten minutes.

Worked example

Mei's summary: "Briefed clients reported about 2.6 times as many suspicious emails per 100 staff, but briefed clients differ in joining date and type. A small randomised follow-up is proposed." Raj approves the follow-up.

Synthetic case. Figures are invented.

Errors and limits

Observational comparisons suggest; randomised comparisons test.

Exercises

Synthesis

Practical research uses comparable measures, recognises confounding and moves toward fairer tests.

Continue studying — chapter 24 of 30

← Chapter 23 (draft) · Course chapter index · Chapter 25 (draft) →