Chapter 33 — Research methods II

Lesson map

Lesson 33 published. Map, example, EX097.

flowchart TD
  N0["Learning outcomes"]
  N1["33.1 Comparative design"]
  N2["33.2 Reliability and validity"]
  N3["33.3 Planning analysis"]
  N4["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4
  N4 --> EX["Worked example"]
  EX --> Q["Exercise EX097"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["33.1 Comparative design"]
  N2["33.2 Reliability and validity"]
  N3["33.3 Planning analysis"]
  N4["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX097"]

[Learning outcomes]
    |
    v
[33.1 Comparative design]
    |
    v
[33.2 Reliability and validity]
    |
    v
[33.3 Planning analysis]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX097]

Learning outcomes

You will (1) design a small comparative study; (2) consider reliability and validity; (3) plan analysis before collecting data.

33.1 Comparative design

For her final year, Nadia plans to compare Ashcombe with another fictional library network, Dunmore Libraries, which has a compliance-led programme of annual training and monthly simulations. Her question is whether the reporting-centred approach at Ashcombe is associated with faster reporting of real suspicious emails than Dunmore's approach. She plans to collect anonymised reporting data and run the same short survey at both. A comparative design cannot prove causation, but it can show whether differences are large enough to merit further attention.

33.2 Reliability and validity

Reliability means a measure gives consistent results; validity means it measures what it should. Nadia checks that both networks define reports the same way and that their IT teams record times similarly. She uses the same survey wording. She notes threats to validity: the networks differ in size, location and the attacks they receive. She plans to describe these differences openly rather than ignore them, and to treat any finding as an indication rather than a conclusion about which programme is better.

33.3 Planning analysis

Before collecting data, Nadia writes her analysis plan: compare median time to first report, compare the share of real malicious emails reported and compare survey scores on confidence and fear of blame. She specifies which tests she will use and states what difference she would consider meaningful. Writing the plan first prevents her from searching the data for whatever result looks interesting. Dr Prescott approves the plan and both networks agree to share anonymised data under a data sharing agreement. The plan is two pages long.

Worked example

Nadia designs a comparison between Ashcombe and Dunmore, addresses reliability and validity and writes an analysis plan in advance.

Synthetic case. Both networks are synthetic.

Errors and limits

Comparisons between organisations are confounded. Recording practices differ.

Exercises

• EX097 — Dunmore approach. What approach does Dunmore use?
  Show marking criteria

  Rubric: 3 = annual training and monthly simulations;; 2 = compliance-led;; 1 = reporting-led;; 0 = none.
• EX098 — Reliability definition. What does reliability mean?
  Show marking criteria

  Rubric: 3 = consistent results from a measure;; 2 = consistency;; 1 = validity;; 0 = none.
• EX099 — Plan first. Why write the analysis plan before data?
  Show marking criteria

  Rubric: 3 = to avoid searching for interesting results;; 2 = pre-planning;; 1 = vague;; 0 = none.

Synthesis

Comparative studies need consistent measures, open threats and analysis planned in advance.

Continue studying — chapter 33 of 40

← Chapter 32 (draft) · Course chapter index · Chapter 34 (draft) →