Chapter 33 — Research methods II
Lesson map
Lesson 33 published. Map, example, EX097.
flowchart TD
N0["Learning outcomes"]
N1["33.1 Returning to the study"]
N2["33.2 Quantitative analysis"]
N3["33.3 Qualitative coding"]
N4["33.4 Validity"]
N5["33.5 Reliability"]
N6["33.6 Triangulation"]
N7["33.7 Reporting"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX097"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["33.1 Returning to the study"]
N2["33.2 Quantitative analysis"]
N3["33.3 Qualitative coding"]
N4["33.4 Validity"]
N5["33.5 Reliability"]
N6["33.6 Triangulation"]
N7["33.7 Reporting"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX097"]
[Learning outcomes]
|
v
[33.1 Returning to the study]
|
v
[33.2 Quantitative analysis]
|
v
[33.3 Qualitative coding]
|
v
[33.4 Validity]
|
v
[33.5 Reliability]
|
v
[33.6 Triangulation]
|
v
[33.7 Reporting]
|
v
[Worked example] --> [Exercise EX097]
Learning outcomes
You will (1) analyse quantitative and qualitative data from a SOC study; (2) use appropriate statistics and coding; (3) assess validity and reliability; (4) report findings with limits.
33.1 Returning to the study
Lucas's survey on student reporting (chapter 18) received 412 responses, and he conducted five interviews.
The responses came through the several channels planned in chapter 18, and Lucas recorded which faculty each respondent studied in. Five interviews are few, but they were meant to explain reasons rather than measure how common they are, which the survey does. Each interview lasted about twenty minutes and was transcribed with identifying details removed. Interviewees were recruited from survey respondents who left an address for follow-up, chosen from different faculties so that the five conversations would not all reflect one department's habits or one course's teaching about phishing.
33.2 Quantitative analysis
61% of respondents said they would delete a suspicious email without reporting; 24% would report; 15% were unsure. With 412 responses, the margin of error is roughly ±5 percentage points, assuming responses are representative.
The margin applies to each percentage separately and assumes a random sample, which a volunteer survey is not. Lucas therefore presents it as a rough guide to precision, not a guarantee. The three answers add to 100%, which confirms that every respondent chose exactly one option and none were lost in coding.
33.3 Qualitative coding
Interview transcripts are coded into themes: "didn't know how to report" (four of five), "thought IT already knew" (three), "worried about looking foolish" (two). A second student codes two transcripts independently; they agree on most codes and discuss differences.
Counts such as four of five show how often a theme came up among interviewees, not how common it is among all students; the survey does that job. Where the two coders disagreed, it was usually about whether "I wasn't sure who to tell" meant not knowing how to report, and they settled on including it.
33.4 Validity
Volunteers may be more security-aware than average, so true reporting may be even lower. Self-reported intentions may differ from actual behaviour; log data on actual reports helps triangulate.
Lucas compares the survey's faculty mix with enrolment: computing students were over-represented, which supports the concern that respondents were more aware than average. Rather than weighting the results, which needs assumptions he cannot check, he reports the imbalance and the direction in which it probably pushes the findings. Self-report is the second concern, and the log comparison below addresses it as far as the data allow.
33.5 Reliability
Clear code definitions and double coding make qualitative results more reliable.
Lucas's codebook defines each theme with an example quote and a counter-example, so the second coder knew what to exclude. After their discussion, they rewrote one definition and recoded the two shared transcripts, reaching agreement on all codes. A reader can repeat the coding using the codebook in the appendix. Reliability is not the same as validity: two coders can agree consistently on a theme that does not reflect what interviewees meant. Lucas guarded against that by checking each theme against the full transcripts, not only the coded extracts.
33.6 Triangulation
Actual reporting rates from chapter 1 (3.0 per 1,000 students) are consistent with low reporting intentions.
The two sources measure different things, intentions and actual behaviour, and both point the same way. Neither alone would be convincing: intentions may not match action, and the reporting rate does not explain why. Together, they support the conclusion that low reporting reflects not knowing how, which the interviews suggested. Lucas is careful not to overstate the match. The survey asked about intentions in general, while the reporting rate comes from one campaign, so the comparison shows consistency rather than proof of cause.
33.7 Reporting
The report gives numbers with uncertainty, quotes that illustrate themes, and limits.
Lucas's report presents each percentage with its margin, gives one anonymised quote per theme and lists limits first in a short paragraph rather than burying them at the end. Readers in the SOC wanted to know whether to trust the numbers before reading them, so putting limits early helped them judge. The appendix holds the full questionnaire and codebook. Lucas also gives the number of respondents who skipped each question, since skipped answers can change percentages and readers deserve to see how many responses each figure rests on.
Ethical reporting
Quotes are chosen to illustrate themes without identifying speakers: course names, years and unusual details are removed. Participants were told how quotes would be used before they agreed to take part. Lucas also shares a short summary of findings with participants, as promised in the consent form, before the full report is submitted.
Sharing the instrument
The questionnaire is published as an appendix so others can repeat the survey.
Worked example
Lucas's main finding: "Most students who received suspicious email would delete it without reporting (61%, ±5 points); interviews suggest the main barrier is not knowing how to report." This supports the report button introduced in chapter 24 and suggests adding a one-line explanation in the student newsletter.
Synthetic case. The survey data are invented.
Errors and limits
One university's survey cannot represent students elsewhere.
Exercises
- EX097 — Survey results. Report the three response shares and the margin of error. Rubric: 3 = 61% delete, 24% report, 15% unsure; ±5 points; 2 = partial; 1 = one; 0 = none.
- EX098 — Main theme. What was the most common interview theme? Rubric: 3 = did not know how to report (4 of 5); 2 = partial; 1 = other; 0 = none.
- EX099 — Validity threat. Name one validity threat and its likely direction. Rubric: 3 = volunteers more aware → true reporting even lower; or intentions ≠ behaviour; 2 = partial; 1 = vague; 0 = none.
Synthesis
Analyse mixed data carefully, triangulate with logs and report findings with honest limits.
Continue studying — chapter 33 of 40
← Chapter 32 (draft) · Course chapter index · Chapter 34 (draft) →