Chapter 3 — Study design, ethics and reproducibility
Lesson map
Lesson 03 published. Map, example, EX015.
flowchart TD
N0["Learning outcomes"]
N1["3.1 Mixed methods"]
N2["3.2 Ethical approval"]
N3["3.3 Self-report risks"]
N4["3.4 Reproducibility plan"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX015"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["3.1 Mixed methods"]
N2["3.2 Ethical approval"]
N3["3.3 Self-report risks"]
N4["3.4 Reproducibility plan"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX015"]
[Learning outcomes]
|
v
[3.1 Mixed methods]
|
v
[3.2 Ethical approval]
|
v
[3.3 Self-report risks]
|
v
[3.4 Reproducibility plan]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX015]
Learning outcomes
You will (1) design a mixed-methods study of a policy change; (2) obtain ethical approval for staff research; (3) plan for reproducibility.
3.1 Mixed methods
Sofia designs a study with three parts. First, she analyses monthly help desk password reset counts for eighteen months before and twelve months after the policy change, an interrupted time series that checks whether the level or trend changed. Second, she surveys staff about password habits, including reuse and writing passwords down, using anonymous questions. Third, she interviews twelve staff from different roles about how the change affected their work. Combining counts, survey answers and interviews lets each method cover weaknesses in the others.
3.2 Ethical approval
Sofia applies to Harrowgate's ethics committee and to the trust's research office. Her survey collects no names and asks no questions about actual passwords; it asks only about habits. Interviewees give written consent, can withdraw and are quoted only by role. Help desk data is provided as monthly counts without any names. The committee asks her to avoid recruiting her own direct colleagues to interviews, because they might feel unable to refuse. She agrees and recruits through a neutral notice instead. All consent forms are stored securely at Harrowgate.
3.3 Self-report risks
Asking staff whether they reuse passwords invites socially desirable answers. Sofia uses wording that normalises the behaviour, such as noting that many people reuse passwords for convenience, and asks about frequency rather than yes or no. She keeps the survey anonymous and avoids any link to individual accounts. She still expects reuse to be under-reported and plans to compare before and after answers rather than treating the absolute level as accurate. Knowing a measure's bias lets her interpret it sensibly. She explains this choice in her methods chapter.
3.4 Reproducibility plan
Sofia writes an analysis plan before she sees the after-change data. She stores code, the survey instrument, the interview guide and anonymised data in a versioned repository. Interview transcripts are not shared because they could identify staff; instead, she shares her coding frame and counts. Dr Nakamura reminds her that reproducibility in human research means sharing everything that can be shared safely and explaining clearly what cannot be shared and why, rather than either sharing nothing or risking participants' privacy. She lists the shared items in a short data statement.
Worked example
Sofia designs a time series, survey and interview study, obtains approval with safeguards, reduces self-report bias and writes a reproducibility plan.
Synthetic case. The study and committees are synthetic.
Errors and limits
Interrupted time series can be confounded by other changes at the same time. Surveys under-report sensitive habits.
Exercises
- EX015 — Three parts. What are the three parts of Sofia's study? Rubric: 3 = reset counts time series, anonymous survey and interviews;; 2 = two;; 1 = vague;; 0 = none.
- EX016 — Survey limits. What does the survey not ask? Rubric: 3 = names or actual passwords;; 2 = passwords;; 1 = vague;; 0 = none.
- EX017 — Colleague recruitment. Why avoid recruiting direct colleagues? Rubric: 3 = they might feel unable to refuse;; 2 = pressure;; 1 = vague;; 0 = none.
- EX018 — Normalising wording. How does Sofia reduce socially desirable answers? Rubric: 3 = wording that normalises reuse and asks frequency;; 2 = wording;; 1 = vague;; 0 = none.
- EX019 — Comparing levels. Why compare before and after rather than trust absolute levels? Rubric: 3 = reuse is likely under-reported consistently;; 2 = relative;; 1 = absolute;; 0 = none.
- EX020 — Unshared transcripts. What is shared instead of interview transcripts? Rubric: 3 = the coding frame and counts;; 2 = coding frame;; 1 = transcripts;; 0 = none.
Synthesis
Mixed methods with careful ethics and an honest reproducibility plan suit research on staff security behaviour.
Continue studying — chapter 03 of 08
← Chapter 02 (draft) · Course chapter index · Chapter 04 (draft) →