OpenCyber Academy
Doctoral research · Chapter 3 · English
Course code: OCA-199Chapter 03: OCA-199-C003

Chapter 3 — Original contribution proposal

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 03 published. Map, example, EX015.

flowchart TD
  N0["Learning outcomes"]
  N1["3.1 What counts as original"]
  N2["3.2 Supporting work"]
  N3["Lab — replaying detection conditions"]
  N4["3.3 Reading the result"]
  N5["3.4 Novelty"]
  N6["3.5 Feasibility"]
  N7["3.6 Risks"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX015"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["3.1 What counts as original"]
  N2["3.2 Supporting work"]
  N3["Lab — replaying detection conditions"]
  N4["3.3 Reading the result"]
  N5["3.4 Novelty"]
  N6["3.5 Feasibility"]
  N7["3.6 Risks"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX015"]

[Learning outcomes]
    |
    v
[3.1 What counts as original]
    |
    v
[3.2 Supporting work]
    |
    v
[Lab — replaying detection conditions]
    |
    v
[3.3 Reading the result]
    |
    v
[3.4 Novelty]
    |
    v
[3.5 Feasibility]
    |
    v
[3.6 Risks]
    |
    v
[Worked example] --> [Exercise EX015]

Learning outcomes

You will (1) state an original contribution precisely; (2) separate the contribution from its supporting work; (3) evaluate detection conditions against labelled synthetic events; (4) defend the proposal's novelty and feasibility.

3.1 What counts as original

An original contribution adds knowledge others can use and test: a validated method, a measured effect, a reusable dataset. Nadia proposes three items: a labelled synthetic alert corpus with benign triggers; a controlled experiment on context enrichment; and an analysis of where enrichment fails. Each item can be checked separately by others. The corpus can be reused for different experiments; the measured effect can be challenged by repeating the study; and the failure analysis gives designers concrete cases. Nadia's contribution statement names the three items so examiners can judge each one against its evidence.

3.2 Supporting work

Detection rules must be good enough that triage, not detection, is the bottleneck in the experiment. Nadia evaluates candidate rules on labelled synthetic events before building alert cards. Rules are written in the Sigma format from SigmaHQ so others can read them in a vendor-neutral way. The rule evaluation is supporting work, not a contribution in itself. Its purpose is to make sure that the alerts in the experiment come from plausible rules, including one that fires on benign activity, so that triage decisions are tested on realistic material.

Lab — replaying detection conditions

"""Replay a Sigma-style detection condition over synthetic labelled events."""
events = [  # (process, parent, cmdline_has_encoded, label)
    ("powershell.exe", "winword.exe", True, "attack"),
    ("powershell.exe", "explorer.exe", False, "benign"),
    ("powershell.exe", "sccm_agent.exe", True, "benign"),
    ("powershell.exe", "excel.exe", True, "attack"),
    ("cmd.exe", "winword.exe", False, "attack"),
    ("powershell.exe", "explorer.exe", True, "benign"),
    ("powershell.exe", "outlook.exe", True, "attack"),
    ("powershell.exe", "sccm_agent.exe", True, "benign"),
]
office = {"winword.exe", "excel.exe", "outlook.exe"}
def v1(p, par, enc): return p == "powershell.exe" and enc
def v2(p, par, enc): return p in {"powershell.exe", "cmd.exe"} and par in office
for name, rule in (("v1 encoded-only", v1), ("v2 office-parent", v2)):
    tp = sum(rule(p, q, e) and l == "attack" for p, q, e, l in events)
    fp = sum(rule(p, q, e) and l == "benign" for p, q, e, l in events)
    fn = sum((not rule(p, q, e)) and l == "attack" for p, q, e, l in events)
    print(f"{name:17} TP={tp} FP={fp} FN={fn} precision={tp/(tp+fp):.2f} recall={tp/(tp+fn):.2f}")

Run with python3 lab_replay.py. Verified output:

v1 encoded-only   TP=3 FP=3 FN=1 precision=0.50 recall=0.75
v2 office-parent  TP=4 FP=0 FN=0 precision=1.00 recall=1.00

3.3 Reading the result

Condition v1 (encoded PowerShell command line) catches 3 of 4 attacks but also fires on 3 benign events — the software-deployment agent and an administrator — giving precision 0.50. Condition v2 (script host whose parent is an Office application) catches all 4 attacks with no false alarms in this small set. The benign triggers of v1 are exactly the kind of alerts context should help analysts dismiss, so Nadia keeps v1's alerts in the corpus as triage material. Both results come from a small labelled set, so Nadia does not claim v2 is a better rule in general; she only checks that each rule behaves as intended on known events. The administrator and deployment-agent alerts give her benign triggers whose context, such as user role or a history of benign closures, is exactly what enriched cards display.

3.4 Novelty

Her review shows prior work measured analysts' perceptions (Alahmadi et al., 2022) and automated ranking (Hassan et al., 2019), but few controlled experiments on context in alert cards with known ground truth. The gap is narrow but real: practitioners report that context helps, and ranking systems reduce volume, yet few studies measure whether showing context changes analysts' decisions on alerts with known truth. Nadia's literature map supports this claim, and she keeps the search record so examiners can check it. The search covered security and human-factors venues, because studies of analyst decisions are often published outside security conferences.

3.5 Feasibility

Eight analysts from the fictional Northgate Shared SOC, which serves several municipalities, agree to participate in work time with manager Ruth Okonkwo's approval; the corpus is synthetic; the study fits in two semesters. Running sessions in work time with the manager's approval avoids asking analysts to give up their own time, but it also means that urgent incidents could interrupt a session. The protocol allows a session to be paused and resumed, with the pause recorded. Two semesters allow for the pilot, the main study and the analysis with some slack.

3.6 Risks

Recruitment may fall short; analysts may recognise synthetic patterns. Mitigations: a reserve pool from a second SOC; realism review by two senior analysts. Recognising synthetic patterns could inflate accuracy in both conditions, which would narrow any difference. The realism review checks the corpus before the main study; the second SOC's reserve pool is contacted only if Northgate's volunteers fall below eight. Both mitigations are recorded in the proposal with who is responsible. Nadia reviews the risk list with Prof. Grant each month until the main study ends.

Worked example

Nadia's contribution statement: "C1: an openly licensed synthetic alert corpus of 320 alerts with labels and benign-trigger annotations. C2: a crossover experiment estimating the effect of context enrichment on triage accuracy and time. C3: a failure analysis of alerts where enrichment misled analysts."

Synthetic case. Events and rules are from the lab script; Northgate is invented.

Errors and limits

Eight events cannot validate a rule; the lab shows method, not rule quality.

Exercises

Synthesis

State contributions precisely, make supporting work fit for purpose, and show novelty and feasibility with evidence.

Continue studying — chapter 03 of 08

← Chapter 02 (draft) · Course chapter index · Chapter 04 (draft) →

Worked answers

EX015. An open labelled synthetic alert corpus, a crossover experiment on context enrichment, and a failure analysis of misleading enrichment.

EX016. If rules were poor, detection errors would swamp the triage effect the study is designed to measure.

EX017. v1 has precision 0.50 and recall 0.75; v2 has precision 1.00 and recall 1.00 on this set.

EX018. They are realistic legitimate matches, the very cases enrichment should help analysts close correctly.

EX019. Earlier studies measured perceptions or automated ranking, but few ran controlled experiments on alert-card context with known labels.

EX020. Whether either rule is valid in general; eight events only demonstrate how replay scoring works.