◇ OpenCyber Academy
Bachelor's · Chapter 33 · English

Chapter 33 — Research methods II

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 33 — Research methods II

Lesson map

Lesson 33 published. Map, example, EX097.

flowchart TD
  N0["Learning outcomes"]
  N1["33.1 From codes to tables"]
  N2["33.2 Agreement"]
  N3["33.3 Uncertainty"]
  N4["33.4 Methods section"]
  N5["33.5 Open materials"]
  N6["33.6 Negative results"]
  N7["Choosing charts"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX097"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["33.1 From codes to tables"]
  N2["33.2 Agreement"]
  N3["33.3 Uncertainty"]
  N4["33.4 Methods section"]
  N5["33.5 Open materials"]
  N6["33.6 Negative results"]
  N7["Choosing charts"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX097"]

[Learning outcomes]
    |
    v
[33.1 From codes to tables]
    |
    v
[33.2 Agreement]
    |
    v
[33.3 Uncertainty]
    |
    v
[33.4 Methods section]
    |
    v
[33.5 Open materials]
    |
    v
[33.6 Negative results]
    |
    v
[Choosing charts]
    |
    v
[Worked example] --> [Exercise EX097]

Learning outcomes

You will (1) analyse coded data with simple statistics; (2) measure inter-coder agreement; (3) present results with uncertainty; (4) write a methods section others can replicate.

33.1 From codes to tables

After coding reports, build a table of technique counts by period or sector. Present counts and denominators. A simple bar chart works if axes are labelled and the sample size is visible.

Amara adds the denominator to every bar label and keeps the axes starting at zero, so small differences are not exaggerated. Her table for one quarter shows phishing in most reports, but with only a few reports per month the monthly bars jump around. She shows the quarter as a whole instead and explains why.

33.2 Agreement

Percentage agreement between two coders is easy but inflated by chance. Cohen's kappa corrects for agreement expected by chance. Values near 0 mean chance-level agreement; values above roughly 0.6 are often treated as substantial, though thresholds are conventions.

Amara reports both figures for her coding. Her two coders agree on most reports, but the kappa is lower than the raw agreement suggests, because most reports contain phishing and agreeing on it is easy. Reporting both shows readers how much of the agreement reflects the coding scheme and how much reflects the sample. Codes with low agreement are discussed and their definitions rewritten.

33.3 Uncertainty

With small samples, report ranges or say "too few to compare". Do not run significance tests on a handful of reports and present them as findings.

Amara's quarterly comparison has fewer than ten reports per period, so she reports counts side by side and states that the difference could easily arise by chance. She suggests pooling several quarters before drawing conclusions about trends. This honest framing is more useful to HSSC than a test result that would look precise but rest on almost no data.

33.4 Methods section

Describe sources, inclusion criteria, dates, coding scheme, coders, agreement and limitations. A reader should be able to repeat the study with the same public sources.

Amara writes her method paragraph alongside the analysis rather than at the end, so details are recorded while fresh. It names the HSSC and public sources, the date range, the coding scheme with its version, the two coders and the kappa value. Limitations come last and are specific, such as the reliance on reports that members chose to share.

33.5 Open materials

Share coding sheets and scripts where licences allow, removing restricted content. Public sources can be listed; TLP-restricted ones cannot.

Amara publishes her coding sheet for public reports and her scripts, with restricted entries removed. For HSSC reports, she lists only the number used and the period, so readers know the material existed without seeing it. HSSC's coordinator checks the release before it goes out. Other students have since reused her scripts on their own samples. Licences are checked before release.

33.6 Negative results

A study that finds no difference between periods is still informative if the method was sound. Report it plainly; hiding unexciting results distorts the literature and wastes the next student's effort repeating the same work.

Amara's own comparison of two quarters found no clear change in phishing frequency. She reports the result in full, with the method and counts, and notes that the sample was too small to detect a modest change. The finding is still useful: it tells HSSC that a single quarter's rise in phishing reports should not be read as a trend, and it gives the next student a baseline to compare against.

Choosing charts

Bar charts suit category counts; line charts suit change over time; tables suit small datasets where exact numbers matter. Avoid pie charts with many slices and three-dimensional effects that distort size. Every chart needs a title, labelled axes, the sample size and the source, so a reader can judge it without the surrounding text. When counts are tiny, a table is usually more honest than any chart.

Worked example

Amara and a classmate code 30 public health-sector incident reports. They agree on 24 of 30 entry-route labels (80%), but kappa is 0.62 because phishing dominates, inflating chance agreement. They report both, discuss the six disagreements and refine two definitions.

Synthetic case. The study figures are invented for teaching.

Errors and limits

Kappa depends on category prevalence. Report the confusion table alongside it.

Exercises

Synthesis

Tabulate honestly, measure agreement with chance correction, respect uncertainty and write replicable methods.

Continue studying — chapter 33 of 40

← Chapter 32 (draft) · Course chapter index · Chapter 34 (draft) →