◇ OpenCyber Academy
Master's · Chapter 6 · English

Chapter 6 — Data collection and analysis

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 6 — Data collection and analysis

Lesson map

Lesson 06 published. Map, example, EX033.

flowchart TD
  N0["Learning outcomes"]
  N1["6.1 Collection"]
  N2["6.2 Primary analysis"]
  N3["6.3 Secondary analyses"]
  N4["6.4 Missingness and exclusions"]
  N5["6.5 Timeline discipline"]
  N6["6.5b Handling exclusions honestly"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX033"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["6.1 Collection"]
  N2["6.2 Primary analysis"]
  N3["6.3 Secondary analyses"]
  N4["6.4 Missingness and exclusions"]
  N5["6.5 Timeline discipline"]
  N6["6.5b Handling exclusions honestly"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX033"]

[Learning outcomes]
    |
    v
[6.1 Collection]
    |
    v
[6.2 Primary analysis]
    |
    v
[6.3 Secondary analyses]
    |
    v
[6.4 Missingness and exclusions]
    |
    v
[6.5 Timeline discipline]
    |
    v
[6.5b Handling exclusions honestly]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX033]

Learning outcomes

You will (1) collect ticket data without breaking blinding; (2) analyse primary and secondary measures with uncertainty; (3) report tag-audit quality; (4) separate confirmatory from exploratory analyses; and (5) handle missingness and exclusions transparently.

6.1 Collection

Export tickets weekly with format flag, severity, asset class, timestamps, and closed-reason. Do not tell remediators which arm is "expected to win". Assessors who write findings should not grade remediation.

Weekly exports let Noah spot missing fields early, without looking at results by arm. Keeping remediators unaware of the expected winner protects against the Hawthorne effect named in Chapter 4. Separating writers from graders prevents an assessor from judging their own findings more kindly. Store each export with its date and hash.

6.2 Primary analysis

Compare 30-day remediation proportions with intervals. Prefer a risk difference or risk ratio with a confidence interval. Stratify or adjust for severity if randomisation used strata.

"""Lab S: reporting attack-success rates honestly: Wilson intervals,
seed variation and a paired bootstrap. Synthetic trial outcomes only."""
import math, random
def wilson(k, n, z=1.96):
    p = k/n; d = 1 + z*z/n
    c = (p + z*z/(2*n))/d; h = z*math.sqrt(p*(1-p)/n + z*z/(4*n*n))/d
    return c - h, c + h
for k, n in ((3, 20), (30, 200), (300, 2000)):
    lo, hi = wilson(k, n); print(f"successes={k}/{n} rate={k/n:.3f} wilson95=[{lo:.3f}, {hi:.3f}]")
# five seeds of a synthetic defence evaluation: per-seed attack success rate
rates = []
for seed in range(5):
    r = random.Random(seed); rates.append(sum(r.random() < 0.12 for _ in range(200))/200)
m = sum(rates)/5; sd = math.sqrt(sum((x-m)**2 for x in rates)/4)
print("per-seed rates:", " ".join(f"{x:.3f}" for x in rates), f"mean={m:.3f} sd={sd:.3f}")
# paired bootstrap: baseline vs defence on the same 300 synthetic trials
r = random.Random(42)
base = [r.random() < 0.30 for _ in range(300)]
defn = [b and r.random() < 0.6 for b in base]   # defence blocks ~40% of successes
diffs = []
for _ in range(2000):
    idx = [r.randrange(300) for _ in range(300)]
    diffs.append(sum(base[i] for i in idx)/300 - sum(defn[i] for i in idx)/300)
diffs.sort()
print(f"baseline_asr={sum(base)/300:.3f} defended_asr={sum(defn)/300:.3f} "
      f"reduction_ci95=[{diffs[50]:.3f}, {diffs[1949]:.3f}]")

Expected output:

successes=3/20 rate=0.150 wilson95=[0.052, 0.360]
successes=30/200 rate=0.150 wilson95=[0.107, 0.206]
successes=300/2000 rate=0.150 wilson95=[0.135, 0.166]
per-seed rates: 0.110 0.130 0.135 0.105 0.145 mean=0.125 sd=0.017
baseline_asr=0.350 defended_asr=0.190 reduction_ci95=[0.120, 0.203]

Lab S shows the habit with synthetic trials: 3 successes in 20 give a Wilson interval from 0.052 to 0.360, while 300 in 2,000 give 0.135 to 0.166. The same rate means very different certainty, so report the counts and the interval beside every proportion. For Noah, the primary result is a risk difference of about 18 points between 62% of 110 tagged and 44% of 108 narrative findings, with an interval that excludes zero. Severity was a stratum, so he also reports the difference within each severity level.

6.3 Secondary analyses

Time-to-first-response; tag-audit agreement; completion by asset class. Label secondary analyses clearly. Do not promote a secondary result to "the finding" if the primary is null.

For Noah, the secondary analyses help explain the primary result. Tag-audit agreement shows how reliable the intervention was; time-to-first-response shows whether tagged tickets were picked up sooner; and asset-class results show where any effect was largest. Each is reported with its interval and labelled secondary or exploratory. If the primary result had been null, none of these would have become the headline.

6.4 Missingness and exclusions

Pre-register rules: tickets reopened within 30 days count as not lastingly remediated; tickets on decommissioned synthetic assets are excluded and listed. Report exclusion counts.

Noah applies both rules from a script written before collection, so a reopened ticket moves to the not-remediated column without any judgement by the analyst. The VLAN 40 rebuild week falls under a pre-registered infrastructure exclusion, and the results table shows the difference with and without it, 18 and 16 points. Exclusion counts appear in the results table, not only in an appendix. Readers should be able to check that no other tickets were dropped and that the excluded week did not change the conclusion.

6.5 Timeline discipline

Keep an operations timeline of assessments, exports and incidents (even synthetic training incidents) so analysis can be reconstructed.

"""Rebuild a synthetic engagement timeline from event records."""
events = [
    ("09:00", "ROE brief signed", "auth"),
    ("09:15", "scoped host inventory loaded (12 hosts)", "prep"),
    ("10:02", "service enumeration on 10.20.1.0/28", "enum"),
    ("10:45", "finding SYN-001 noted on staging UI", "find"),
    ("11:10", "stop: host 10.20.0.1 probed — out of scope", "stop"),
    ("11:12", "escalated to client contact; confirmed deny-list", "comm"),
    ("13:00", "resume on in-scope hosts only", "auth"),
    ("15:30", "debrief draft shared with client", "report"),
]
# print chronological with phase tags; compute stop latency
print("phase timeline:")
for t, msg, phase in events:
    print(f"  {t} [{phase:6}] {msg}")
t0 = 10*60+45; t1 = 11*60+10
print(f"minutes from SYN-001 note to stop: {t1-t0}")
print(f"stop reason present: yes")
print(f"events={len(events)} phases={len(set(p for _,_,p in events))}")

Expected output:

phase timeline:
  09:00 [auth  ] ROE brief signed
  09:15 [prep  ] scoped host inventory loaded (12 hosts)
  10:02 [enum  ] service enumeration on 10.20.1.0/28
  10:45 [find  ] finding SYN-001 noted on staging UI
  11:10 [stop  ] stop: host 10.20.0.1 probed — out of scope
  11:12 [comm  ] escalated to client contact; confirmed deny-list
  13:00 [auth  ] resume on in-scope hosts only
  15:30 [report] debrief draft shared with client
minutes from SYN-001 note to stop: 25
stop reason present: yes
events=8 phases=7

The timeline lab rebuilds a synthetic engagement from eight events, including the stop when out-of-scope host 10.20.0.1 was probed and the escalation two minutes later. For Noah's study, the timeline records assessment windows, weekly exports, the VLAN 40 rebuild and authorization renewals, so examiners can see when each event happened relative to the data. Record each event as it happens, with its source.

6.5b Handling exclusions honestly

The VLAN 40 rebuild week is the chapter's lesson in exclusions. An exclusion is legitimate only when it was named in the pre-registration with its trigger ("infrastructure rebuild affecting the test segment"), when it is applied blind to the outcome, and when the report shows the result with and without it. Here the 18-point difference becomes 16 points if the week is kept; both numbers appear in the results table, and the conclusion does not depend on the choice. An exclusion invented after seeing the data, or one that flips the conclusion, must be reported as a limitation rather than silently applied.

6.6 Worked example (synthetic CBM case)

Main collection: 110 tagged and 108 narrative findings. Remediated in 30 days: 62% vs 44% (difference about 18 points; interval excludes 0). Tag-audit agreement 96% after second-person review. Exploratory: largest lift on medium-severity web findings. One week excluded when VLAN 40 was rebuilt — pre-registered infra exclusion.

The primary result is reported with its interval and conditions. The tag audit shows that the intervention was applied correctly in most tickets after the second-person check. The medium-severity web lift is labelled exploratory, because it was not pre-registered. The excluded week is named in the results table and limitations, with the result shown both ways.

6.7 Errors and limits

Each error biases the result. Peeking weekly and stopping when the difference looks good inflates the chance of a false finding. Silent exclusions let the researcher shape the data without readers knowing. Treating exploratory subgroup lifts as confirmatory turns one of many comparisons into a headline. Check the analysis plan and results chapter for all three before writing. Noah's analysis plan, written before collection, addresses all three.

Exercises

Synthesis

Collection protects blinding; analysis respects the pre-registered primary measure and tells the truth about exclusions and uncertainty.

Continue studying — chapter 06 of 08

← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →