Chapter 18 — Research methods I | Bachelor's Security operations and blue team

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 18 — Research methods I

Lesson map

Lesson 18 published. Map, example, EX052.

flowchart TD
  N0["Learning outcomes"]
  N1["18.1 From curiosity to question"]
  N2["18.2 Choosing a method"]
  N3["18.3 Sampling"]
  N4["18.4 Ethics"]
  N5["18.5 Designing questions"]
  N6["18.6 Analysis"]
  N7["18.7 Literature"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX052"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["18.1 From curiosity to question"]
  N2["18.2 Choosing a method"]
  N3["18.3 Sampling"]
  N4["18.4 Ethics"]
  N5["18.5 Designing questions"]
  N6["18.6 Analysis"]
  N7["18.7 Literature"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX052"]

[Learning outcomes]
    |
    v
[18.1 From curiosity to question]
    |
    v
[18.2 Choosing a method]
    |
    v
[18.3 Sampling]
    |
    v
[18.4 Ethics]
    |
    v
[18.5 Designing questions]
    |
    v
[18.6 Analysis]
    |
    v
[18.7 Literature]
    |
    v
[Worked example] --> [Exercise EX052]

Learning outcomes

You will (1) formulate a research question about SOC practice; (2) choose methods such as surveys, interviews and log analysis; (3) consider ethics in research with people and data; (4) plan a small study.

18.1 From curiosity to question

Lucas wonders why students report phishing less than staff (chapter 1). A research question: "What reasons do Riverton students give for not reporting suspicious emails?"

The question focuses on reasons, which logs cannot show, and on Riverton students, which keeps the study manageable. It avoids assuming students are careless; perhaps they did not know how to report, or assumed someone else would. A good question leaves room for answers the researcher did not expect, and that openness shapes the methods chosen next.

18.2 Choosing a method

Surveys reach many people with fixed questions; interviews explore reasons in depth with fewer people; log analysis shows behaviour but not reasons. Lucas combines a short survey with five interviews.

The survey gives a sense of how common each reason is, while the interviews explain reasons the survey's options might miss. Log analysis would add little here, since chapter 1's rates already show the behaviour. Combining methods lets each cover the other's weakness, which the embedded SOC studies cited below also recommend.

18.3 Sampling

A survey of volunteers may over-represent security-interested students. Lucas recruits through several channels and reports who responded.

Lucas posts the survey on the student portal, in two faculty newsletters and through a residence-hall mailing list, rather than only on the security society's channel. He reports how many responded from each faculty, so readers can see whether some groups dominate. A sample of mostly computing students would describe them well and the rest of campus poorly. He also compares respondents' year of study with enrolment figures.

18.4 Ethics

Research with people needs ethics approval, informed consent, anonymity and the right to withdraw. No SOC data about individuals is used.

Lucas applies to the university's ethics committee with his questions, consent form and data plan. Separating the study from SOC data is important: his position as an analyst gives him access to logs, but using them for research would breach the purpose limitation described in the privacy chapter. Respondents are told he works in the SOC, and that nothing they say will be passed to it.

18.5 Designing questions

Neutral wording ("How do you usually handle suspicious emails?") avoids leading answers ("Why don't you report phishing?").

The leading version assumes students do not report and invites them to justify themselves, which may produce defensive or invented reasons. The neutral version lets them describe what they actually do, including reporting. Lucas also avoids technical terms such as DMARC, since students who do not know them may guess or skip the question. Each question asks about one thing only, and the pilot described below tests whether readers understand them as intended.

18.6 Analysis

Survey answers are counted; interview transcripts are coded into themes, with a second student coding a sample to check consistency.

For the interviews, Lucas codes themes such as not knowing how to report, assuming IT already knows and fear of looking foolish. A second student codes two of the five transcripts independently, and they compare results. Where they disagree, they discuss and refine the theme definitions, then recode, so the final themes are not just one person's reading of the transcripts.

18.7 Literature

Sundaramurthy et al. (2014) show how researchers embedded in SOCs revealed practice that surveys missed, encouraging mixed methods.

Their finding matters for Lucas's design: analysts in those studies described their work differently in surveys than researchers observed in practice. Students may likewise say they would report suspicious mail while their behaviour in the campaign suggested otherwise. Combining interviews with chapter 1's measured rates lets him compare what students say with what they did, which neither source could show alone or settle.

Analysis plan

Before collecting responses, Lucas writes down how he will analyse them: which questions are counted, how themes will be coded and how disagreements between coders will be resolved. Writing the plan first prevents him from searching the data for whatever looks interesting and presenting it as if it had been the question all along.

Worked example

Lucas's pilot survey with ten classmates shows the question "Do you know how to report phishing?" was misread as "Do you know what phishing is?" He rewrites it: "If you received a suspicious email at Riverton, what would you do with it?" with open text.

Synthetic case. The study is invented.

Errors and limits

Small studies at one university show local reasons; others may differ.

Exercises

EX052 — Method mix. Why combine a survey with interviews? Rubric: 3 = survey breadth plus interview depth on reasons; 2 = partial; 1 = one enough; 0 = none.
EX053 — Leading questions. Rewrite a leading question neutrally. Rubric: 3 = e.g. "How do you usually handle suspicious emails?"; 2 = partial; 1 = unchanged; 0 = none.
EX054 — Pilot finding. What did the pilot reveal and how was it fixed? Rubric: 3 = question misread; rewritten as open scenario; 2 = partial; 1 = vague; 0 = none.

Synthesis

Research methods turn SOC puzzles into questions, chosen methods and ethical, piloted studies.

Continue studying — chapter 18 of 40

← Chapter 17 (draft) · Course chapter index · Chapter 19 (draft) →