◇ OpenCyber Academy
Technologist · Chapter 24 · English

Chapter 24 — Applied research methods

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 24 — Applied research methods

Lesson map

Lesson 24 published. Map, example, EX070.

flowchart TD
  N0["Learning outcomes"]
  N1["24.1 Practical questions"]
  N2["24.2 Data collection"]
  N3["24.3 Coding"]
  N4["24.4 Simple analysis"]
  N5["24.5 Ethics"]
  N6["24.6 Reporting honestly"]
  N7["24.7 Literature"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX070"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["24.1 Practical questions"]
  N2["24.2 Data collection"]
  N3["24.3 Coding"]
  N4["24.4 Simple analysis"]
  N5["24.5 Ethics"]
  N6["24.6 Reporting honestly"]
  N7["24.7 Literature"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX070"]

[Learning outcomes]
    |
    v
[24.1 Practical questions]
    |
    v
[24.2 Data collection]
    |
    v
[24.3 Coding]
    |
    v
[24.4 Simple analysis]
    |
    v
[24.5 Ethics]
    |
    v
[24.6 Reporting honestly]
    |
    v
[24.7 Literature]
    |
    v
[Worked example] --> [Exercise EX070]

Learning outcomes

You will (1) frame a practical research question for a CTI improvement; (2) collect and code data systematically; (3) analyse results simply and honestly; (4) respect research ethics in the workplace.

24.1 Practical questions

"Does the weekly scam list reduce service-desk calls that need escalation?" is answerable with existing data. "Are councils safer with CTI?" is too broad.

The first question names a product, an outcome and data that already exist in service-desk tickets. The second has no single measure and would need data CMS does not have. Lena starts with the narrow question because it can be answered in weeks, and a clear answer to a small question is more useful than a vague answer to a large one.

24.2 Data collection

Use existing records — tickets, logs, feedback forms — where policy allows. Define what counts before collecting: an escalated call means one forwarded to the CTI queue.

Defining an escalated call before counting stops the measure shifting during collection. If some calls were counted because they were long and others because they were forwarded, the before and after numbers would measure different things. Using existing records avoids asking anyone new questions and keeps within what the data's owners already allow. Lena writes the definition into her study notes before collecting, so it cannot quietly change once she sees the data.

24.3 Coding

When reading free-text tickets, use a small set of categories with examples. Code a sample twice, a week apart, to check your own consistency.

A category such as "scam call recognised by caller" needs an example, otherwise the same ticket could be coded differently on different days. Coding a sample twice a week apart shows whether Lena's own coding is stable. If her two passes disagree often, the categories need clearer definitions before the full set is coded. The final category list and examples are kept with the results, so a reader can see how each ticket was classified.

24.4 Simple analysis

Counts before and after, by week, with a chart. Note other changes in the same period, such as a news story about scams, that could explain differences.

A weekly chart shows whether a change appeared suddenly when the scam list started or drifted gradually. In the worked example, total scam calls stayed similar while escalated calls fell from 34 to 21, a drop of 13. The news story about scams during the after period is exactly the kind of other change this rule asks her to record.

24.5 Ethics

Tickets contain residents' and staff's personal data. Use only what the question needs, remove names, store results securely and get the manager's and data-protection officer's approval.

Removing names before analysis means Lena works with categories and counts rather than identifiable tickets. The question only needs to know whether a call was escalated, not who called. The data-protection officer's approval confirms that this use fits the purposes for which the tickets were collected. The anonymised dataset is stored with restricted access and deleted when the study's retention period ends.

24.6 Reporting honestly

State what changed, what else might explain it and how confident you are. A small improvement with honest caveats is more useful than an inflated claim.

For Lena's study, an honest statement says that escalated calls fell from 34 to 21 over eight weeks, that total calls were similar, that a news story may have helped, and that the evidence suggests rather than proves an effect. The manager can then decide whether to continue the list knowing how strong the evidence is.

24.7 Literature

Look for published guidance and studies on security awareness and CTI sharing; cite sources you actually read.

Citing only what was read means that if a claim from a study is questioned, Lena can say exactly where it came from and what it showed. A source cited from someone else's summary may say something different from the summary. Her reference list therefore contains only works she has actually opened and checked. She records where each source was found and when she read it, so the reference list can be checked later.

Choosing a time window

Pick equal periods before and after the change and avoid holidays or known campaign peaks that would distort counts. Eight weeks each side gave Lena enough calls to see a pattern without waiting a whole year. Write the window down before looking at results, so it cannot be adjusted to make the change look better.

Worked example

Lena compares eight weeks before and after starting the scam list. Escalated calls fell from 34 to 21, while total scam-related calls stayed similar. She notes that a regional news story about scams ran during the "after" period, which might have helped callers recognise scams themselves.

Synthetic case. The numbers are invented.

Errors and limits

Before/after comparisons cannot prove causation; they suggest.

Exercises

Synthesis

Ask practical questions, define measures first, code consistently, analyse simply and report with honest caveats.

Continue studying — chapter 24 of 30

← Chapter 23 (draft) · Course chapter index · Chapter 25 (draft) →