Chapter 5 — Proposal and pilot
Lesson map
Lesson 05 published. Map, example, EX027.
flowchart TD
N0["Learning outcomes"]
N1["5.1 The proposal"]
N2["5.2 Piloting the survey"]
N3["5.3 Piloting interviews"]
N4["5.4 Recording revisions"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX027"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["5.1 The proposal"]
N2["5.2 Piloting the survey"]
N3["5.3 Piloting interviews"]
N4["5.4 Recording revisions"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX027"]
[Learning outcomes]
|
v
[5.1 The proposal]
|
v
[5.2 Piloting the survey]
|
v
[5.3 Piloting interviews]
|
v
[5.4 Recording revisions]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX027]
Learning outcomes
You will (1) write a dissertation proposal; (2) pilot the survey and interview guide; (3) revise instruments from pilot evidence.
5.1 The proposal
Sofia's proposal sets out the question, the literature, the theories with their predictions, the three-part design, ethics, the analysis plan and a timeline. It states what would count as support and what would count against: if resets per thousand accounts do not fall at the change point, or if survey reports of writing passwords down rise, she will report the change as not beneficial. Dr Nakamura approves it after she shortens the literature section and adds a risk register for data access delays. The approved proposal is filed with the ethics application.
5.2 Piloting the survey
Sofia pilots the survey with ten staff from a department not in her sample. She times completion and asks each person to explain what they understood by each question. Two problems emerge. Staff interpret "password manager" differently, some thinking of the browser's saved passwords and others of the trust's approved tool. A question about sharing passwords is read as sharing with family by some and with colleagues by others. She rewrites both questions with examples and adds a short definition at the start. Completion takes about eight minutes.
5.3 Piloting interviews
She runs two pilot interviews. The first runs too long because her questions invite general complaints about IT. She reorders the guide to start with a specific recent experience, such as the last time the participant forgot a password, then moves to wider views. The second pilot works better and yields concrete examples. She also learns to pause and let participants finish, rather than filling silences, which produces richer accounts of how password rules interact with clinical work. She records these lessons in her research diary for later reflection.
5.4 Recording revisions
Sofia records every change from the pilot in a revision log with the reason and the evidence that prompted it. Pilot data is not used in the main analysis, which she notes in the plan. Dr Nakamura explains that examiners look for this record because it shows the instruments were tested rather than assumed to work. He also notes that pilots often reveal that researchers and participants use the same words differently, which is especially common in security, where technical terms carry everyday meanings.
Worked example
Sofia writes a proposal with falsifying criteria, pilots the survey and interviews, fixes ambiguous questions and reorders the guide, and logs revisions.
Synthetic case. Pilots and staff are synthetic.
Errors and limits
Small pilots miss rare problems. Revisions after piloting may introduce new ambiguities.
Exercises
- EX027 — Counting against. What results would count against the policy change? Rubric: 3 = no fall in resets per 1,000 or more writing down reported;; 2 = either;; 1 = vague;; 0 = none.
- EX028 — Proposal revision. What did Dr Nakamura ask her to add? Rubric: 3 = a risk register for data access delays;; 2 = risk register;; 1 = vague;; 0 = none.
- EX029 — Ambiguous term. Which term did staff interpret in different ways? Rubric: 3 = password manager;; 2 = manager;; 1 = vague;; 0 = none.
- EX030 — Interview fix. How was the interview guide reordered? Rubric: 3 = start with a specific recent experience, then wider views;; 2 = specific first;; 1 = vague;; 0 = none.
- EX031 — Pilot data use. Is pilot data used in the main analysis? Rubric: 3 = no;; 2 = excluded;; 1 = yes;; 0 = none.
- EX032 — Shared words. What common problem do pilots reveal in security research? Rubric: 3 = researchers and participants use words differently;; 2 = meaning;; 1 = vague;; 0 = none.
Synthesis
Pilots test whether instruments mean what the researcher intends, and logged revisions show examiners they were tested.
Continue studying — chapter 05 of 08
← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →