Chapter 33 — Research methods II

Chapter 33 — Research methods II

Lesson map

Lesson 33 published. Map, example, EX097.

flowchart TD
  N0["Learning outcomes"]
  N1["33.1 Running the study"]
  N2["33.2 Reporting with uncertainty"]
  N3["33.3 Validity threats"]
  N4["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4
  N4 --> EX["Worked example"]
  EX --> Q["Exercise EX097"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["33.1 Running the study"]
  N2["33.2 Reporting with uncertainty"]
  N3["33.3 Validity threats"]
  N4["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX097"]

[Learning outcomes]
    |
    v
[33.1 Running the study]
    |
    v
[33.2 Reporting with uncertainty]
    |
    v
[33.3 Validity threats]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX097]

Learning outcomes

You will (1) analyse data from a small security study; (2) report results with uncertainty; (3) discuss validity threats.

33.1 Running the study

Priya runs the certificate-warning study planned in Chapter 18. Sixty staff are randomly assigned to two groups; one reads a one-page explanation of certificate warnings first. All later see a simulated warning on the training site. In the control group, 18 of 30 proceed past the warning; in the explanation group, 9 of 30 proceed. Data are anonymous and stored according to the ethics approval. Everyone is debriefed afterwards. Priya checks that group sizes and baseline roles are similar across the two groups before comparing results, and records any imbalance she finds.

33.2 Reporting with uncertainty

The proportions are 60 percent and 30 percent. Priya computes the difference, 30 percentage points, and uses a permutation test, shuffling group labels many times, to estimate how often a difference this large would appear by chance; it is rare, under five percent of shuffles. She reports the difference with its uncertainty, noting the small sample. She avoids saying the explanation "works", writing instead that it was associated with fewer staff proceeding in this simulation. She shares the permutation script with the class so others can rerun it.

33.3 Validity threats

Priya discusses threats. Staff in the explanation group might have guessed the study's purpose and behaved differently. A training site differs from real work pressure. Thirty people per group cannot reveal small effects or effects in subgroups. Results come from one consortium. She recommends repeating the study elsewhere and measuring behaviour weeks later to test whether the effect lasts. Honest discussion of threats is part of the result, not an apology for it. Dr. Echeverría asks her to include the threats in the abstract.

Worked example

Priya reports 60 versus 30 percent proceeding past a simulated warning, a permutation result under five percent, and four validity threats.

Synthetic case. The study data are invented.

Errors and limits

Small samples give unstable estimates. Simulated settings may not reflect real behaviour.

Exercises

• EX097 — Study results. Report the share proceeding in each group. Rubric: 3 = 60 percent control (18/30) and 30 percent explanation (9/30);; 2 = one;; 1 = wrong;; 0 = none.
• EX098 — Careful wording. How does Priya describe the effect? Rubric: 3 = associated with fewer staff proceeding in this simulation, not "works";; 2 = careful;; 1 = "works";; 0 = none.
• EX099 — Validity threat. Name two validity threats. Rubric: 3 = two of guessing purpose, simulation vs real pressure, small sample, single consortium;; 2 = one;; 1 = vague;; 0 = none.

Synthesis

Small studies can inform practice when results come with uncertainty and openly discussed threats.

Continue studying — chapter 33 of 40

← Chapter 32 (draft) · Course chapter index · Chapter 34 (draft) →