◇ OpenCyber Academy
Master's · Chapter 6 · English

Chapter 6 — Data collection and analysis

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 6 — Data collection and analysis

Lesson map

Lesson 06 published. Map, example, EX033.

flowchart TD
  N0["Learning outcomes"]
  N1["6.1 Main run"]
  N2["6.2 Results overview"]
  N3["6.3 Agreement study"]
  N4["Lab — examiner agreement"]
  N5["6.4 Interpreting kappa"]
  N6["6.5 Disagreements"]
  N7["6.6 Reporting"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX033"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["6.1 Main run"]
  N2["6.2 Results overview"]
  N3["6.3 Agreement study"]
  N4["Lab — examiner agreement"]
  N5["6.4 Interpreting kappa"]
  N6["6.5 Disagreements"]
  N7["6.6 Reporting"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX033"]

[Learning outcomes]
    |
    v
[6.1 Main run]
    |
    v
[6.2 Results overview]
    |
    v
[6.3 Agreement study]
    |
    v
[Lab — examiner agreement]
    |
    v
[6.4 Interpreting kappa]
    |
    v
[6.5 Disagreements]
    |
    v
[6.6 Reporting]
    |
    v
[Worked example] --> [Exercise EX033]

Learning outcomes

You will (1) run the main study under fixed conditions; (2) measure examiner agreement with Cohen's kappa; (3) analyse reduction against reference currency; (4) report results with counts.

6.1 Main run

Thirty synthetic images, frozen reference set, recorded tool versions. Each image yields reduction, misses and days since reference release. Every image follows the same steps: build from the recorded seed, hash with the validated tool, filter against the frozen NSRL version, and compare results with the labels. Each run writes a log entry with the image, versions and results. Tomás checks a few images by hand to confirm that the scripts count files correctly. The main run uses the revised image builder, which includes common third-party applications.

6.2 Results overview

Mean reduction is high when the OS build precedes the reference release and falls as the gap grows. Designed misses (legitimate tools with planted illegitimate use, vendor documents) are all set aside — filtering cannot distinguish them by hash. The relationship between reduction and the gap to the reference release confirms what the pilot suggested. The designed misses show the limit of hashing: a legitimate tool used for harm and an unmodified vendor document are identical to published files, so no hash comparison can tell them apart from their legitimate copies. Tomás reports the misses by type with their counts, separating legitimate tools used for harm from vendor documents identical to NSRL entries.

6.3 Agreement study

Two examiners classify 40 timeline events from the remaining files as user or system activity. Agreement shows whether the reduced set can be interpreted consistently. Using events from the reduced sets tests whether what remains after filtering can be read consistently by different examiners. The examiners work separately, using a written coding guide, and do not know the study's hypothesis or which files were planted. Their classifications are compared event by event, and each disagreement is recorded with both answers. The examiners come from two units, as planned.

Lab — examiner agreement

"""Cohen's kappa for two examiners classifying 40 synthetic timeline events."""
a = ["user"]*18 + ["system"]*14 + ["user"]*4 + ["system"]*4
b = ["user"]*18 + ["system"]*14 + ["system"]*4 + ["user"]*4
n = len(a); po = sum(x == y for x, y in zip(a, b)) / n
pe = sum((a.count(c)/n) * (b.count(c)/n) for c in ("user", "system"))
print(f"events={n} observed_agreement={po:.2f} expected={pe:.2f} kappa={(po-pe)/(1-pe):.2f}")

Run with python3 lab_kappa.py. Verified output:

events=40 observed_agreement=0.80 expected=0.51 kappa=0.60

6.4 Interpreting kappa

Observed agreement is 0.80, but chance alone would give 0.505 given each examiner's category proportions (the lab prints this rounded to 0.51). Kappa = (0.80 − 0.505) / (1 − 0.505) = 0.596, reported as 0.60, corrects for chance. Landis and Koch (1977) proposed labels in which 0.41–0.60 is "moderate"; Tomás cites the labels as convention, not law. Both examiners used the two categories in the same proportions, 22 user and 18 system events each, so chance alone would produce a match about half the time. Kappa shows how much agreement exceeds that level. Tomás reports observed agreement, chance agreement, kappa and the number of events, so readers can check the calculation and judge the result for themselves.

6.5 Disagreements

The eight disagreements involve scheduled tasks that ran under a user account. The category definition was ambiguous; he records it as a finding for the coding guide. A scheduled task running under a user account can reasonably be read either way: it was set up by a user but runs automatically. The coding guide did not say which category applied, so the examiners each made a sensible choice. Tomás does not count the disagreements as examiner error. He revises the guide to define such tasks explicitly and recommends that future agreement studies test the revised definition.

6.6 Reporting

Tables show counts per image; figures plot reduction against days gap. Every number has a script and a log entry. Plotting reduction against the gap shows the relationship more clearly than an average would. The tables also list misses per image, so readers can see that designed misses were set aside on every image where they were planted. Tomás checks each table against the logs before including it in the thesis. Figures use the same axes for every image group, so they can be compared directly.

6.7 Missing data

One image failed to build; he reports 29 analysed of 30 planned, with the reason. The failed image is reported, not replaced, as the stopping rule required. The build log showed the image builder stopped when installing one application, and the image was never filtered. Tomás records the failure and its cause in the results chapter, so readers know the analysis covers 29 images. He also checks that the failed image was not unusual in any planned variable, so its absence is unlikely to change the results.

Analysis scripts

One script computes reduction and misses per image; another plots reduction against the days gap; a third computes kappa. Each writes its outputs with a hash so tables can be checked against the archive later, and each run is noted in the log with its date and parameters.

Worked example

Tomás drafts: "Kappa was 0.60 (observed 0.80, chance 0.505, n = 40). Disagreements concentrated on scheduled tasks under user accounts; the coding guide will define them explicitly."

Synthetic case. Agreement data come from the lab script.

Errors and limits

Two examiners and 40 events give an imprecise kappa; it is a feasibility estimate.

Exercises

Synthesis

Report the main run with counts, correct agreement for chance, and treat ambiguity as a finding.

Continue studying — chapter 06 of 08

← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →