Chapter 33 — Research methods II | Bachelor's Network and communications security

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 33 published. Map, example, EX097.

flowchart TD
  N0["Learning outcomes"]
  N1["33.1 From question to study design"]
  N2["33.2 Measurement validity"]
  N3["33.3 Reliability of human judgement"]
  N4["33.4 Honest analysis"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX097"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["33.1 From question to study design"]
  N2["33.2 Measurement validity"]
  N3["33.3 Reliability of human judgement"]
  N4["33.4 Honest analysis"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX097"]

[Learning outcomes]
    |
    v
[33.1 From question to study design]
    |
    v
[33.2 Measurement validity]
    |
    v
[33.3 Reliability of human judgement]
    |
    v
[33.4 Honest analysis]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX097]

Learning outcomes

You will (1) design a measurement study; (2) analyse results honestly; (3) check agreement between reviewers.

33.1 From question to study design

For her capstone, Priya asks: after guest separation, does the number of guest-originated connection attempts reaching internal groups fall to zero, and how many guest alerts need human attention each week? The study compares four weeks before and four weeks after deployment at two clinics. She defines each measure precisely, chooses data sources and writes the analysis plan before collecting data, so the analysis cannot be bent to fit the results she hopes to see. Two clinics are chosen because they have similar visitor numbers, which makes their results easier to compare.

33.2 Measurement validity

A measure is valid if it captures what it claims to. "Guest connections reaching internal groups" before deployment comes from flow records, since no firewall existed; after deployment it comes from firewall allow logs. Different sources might count differently, so Priya checks both on a test day when both are available and confirms they agree closely. Without this check, an apparent change could simply reflect a change in how data were counted rather than a change in the network. The agreement check is recorded in her method chapter.

33.3 Reliability of human judgement

Some measures depend on judgement, such as whether an alert needed attention. Two people may judge the same alert differently. Priya and the IT lead each label the same sample of alerts independently, then measure how often they agree beyond what chance would produce. Cohen's kappa is a common statistic for this, and Landis and Koch suggested widely used descriptive bands for interpreting it. Measuring agreement shows whether a judgement-based measure is consistent enough to report. Low agreement would mean the labelling rules need clarifying before results are reported.

33.4 Honest analysis

Priya commits to reporting all results, including those that do not support her design. If guest connections to internal groups do not fall to zero, she will report it and investigate why. She reports sample sizes, the time periods and any data problems. Honest reporting matters especially in student projects with partners, where there can be pressure to present only good news. Partners benefit more from accurate findings than from flattering ones, because accurate findings show what still needs fixing.

Worked example

The analysis plan defines two measures, their sources, a validity check and an agreement check, all written before data collection.

Synthetic case. The study is synthetic.

Errors and limits

Changing measurement sources at the same time as the intervention can create false effects.

Exercises

EX097 — Plan timing. Why write the analysis plan before collecting data?
Show marking criteria
Rubric: 3 = prevents bending analysis to fit results;; 2 = avoid bias;; 1 = tidiness;; 0 = none.
EX098 — Source change risk. Why check flow records against firewall logs on one day?
Show marking criteria
Rubric: 3 = confirm sources count alike, avoid false change;; 2 = validity;; 1 = vague;; 0 = none.
EX099 — Agreement statistic. Which statistic measures agreement between two reviewers beyond chance?
Show marking criteria
Rubric: 3 = Cohen's kappa, Landis and Koch bands;; 2 = kappa;; 1 = percentage only;; 0 = none.

Synthesis

Pre-planned measures, validity checks, agreement statistics and honest reporting make findings trustworthy.

Continue studying — chapter 33 of 40

← Chapter 32 (draft) · Course chapter index · Chapter 34 (draft) →