◇ OpenCyber Academy
Doctoral research · Chapter 3 · English

Chapter 3 — Original contribution proposal

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 3 — Original contribution proposal

Lesson map

Lesson 03 published. Map, example, EX015.

flowchart TD
  N0["Learning outcomes"]
  N1["3.1 Proposal structure"]
  N2["3.2 Defining outcomes"]
  N3["3.3 A small model"]
  N4["Lab — Matrix scores versus expected loss for six sce"]
  N5["3.4 Risks to the study"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX015"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["3.1 Proposal structure"]
  N2["3.2 Defining outcomes"]
  N3["3.3 A small model"]
  N4["Lab — Matrix scores versus expected loss for six sce"]
  N5["3.4 Risks to the study"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX015"]

[Learning outcomes]
    |
    v
[3.1 Proposal structure]
    |
    v
[3.2 Defining outcomes]
    |
    v
[3.3 A small model]
    |
    v
[Lab — Matrix scores versus expected loss for six sce]
    |
    v
[3.4 Risks to the study]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX015]

Learning outcomes

You will (1) write a doctoral proposal for a method comparison; (2) define outcomes for consistency, calibration and decision quality; (3) show with a small model why method choice can change priorities.

3.1 Proposal structure

Amara's proposal has five parts: problem and gap, research questions, design, analysis plan and risks. The research questions are: do calibrated quantitative estimates improve agreement between assessors; do they improve calibration of probability forecasts; and do they change which risks are treated first? The design uses Corvane's own assessors, randomly assigned to method, assessing the same synthetic scenarios built from real patterns. The proposal states expected contributions to theory, to practice and to the evidence base used by regulators. Corvane reviews the draft.

3.2 Defining outcomes

Each outcome has a measure. Consistency is measured by agreement between assessors, using kappa for bands and the ratio of estimates for quantities. Calibration is measured by the Brier score, the average squared difference between forecast probabilities and what actually happened, on short-horizon events that can be checked within the study. Decision quality is measured by whether the treatment order matches the order by expected loss computed from a reference panel. Amara writes these definitions in the pre-registration so they cannot drift later.

3.3 A small model

To show why method choice matters, Amara builds a small example of six synthetic scenarios with annual probabilities and loss sizes. The lab bands each on a five-by-five matrix and also computes expected loss. Ransomware on records has the largest expected loss, about 192 thousand per year, yet the matrix scores it 5, below payment fraud and supplier outage, which tie at 6. This reproduces in miniature the rank reversal and ties that Cox (2008) predicted, and it motivates the study. Prof. Delgado agrees. This gives a clear starting point.

Lab — Matrix scores versus expected loss for six scenarios

# Risk matrix versus expected loss for six synthetic scenarios (after Cox, 2008).
# (name, annual probability, loss in thousands)
S = [("ransomware on records", 0.08, 2400), ("payment fraud", 0.30, 180), ("laptop loss", 0.60, 25),
     ("supplier outage", 0.15, 900), ("insider data leak", 0.05, 1500), ("web defacement", 0.40, 40)]
def band_p(p): return 1 if p < 0.1 else 2 if p < 0.2 else 3 if p < 0.35 else 4 if p < 0.5 else 5
def band_i(x): return 1 if x < 50 else 2 if x < 200 else 3 if x < 1000 else 4 if x < 2000 else 5
rows = [(n, band_p(p) * band_i(x), p * x) for n, p, x in S]
for n, sc, el in sorted(rows, key=lambda t: -t[2]): print(f"{n}: matrix score {sc}, expected loss {el:.0f}k")
by_m = [n for n, _, _ in sorted(rows, key=lambda t: (-t[1], t[0]))]; by_e = [n for n, _, _ in sorted(rows, key=lambda t: -t[2])]
print("top by matrix:", by_m[0], "| top by expected loss:", by_e[0])
print("matrix ties at top score:", sum(1 for _, s, _ in rows if s == max(s for _, s, _ in rows)))

Run with python3 lab_matrix.py. Verified output:

ransomware on records: matrix score 5, expected loss 192k
supplier outage: matrix score 6, expected loss 135k
insider data leak: matrix score 4, expected loss 75k
payment fraud: matrix score 6, expected loss 54k
web defacement: matrix score 4, expected loss 16k
laptop loss: matrix score 5, expected loss 15k
top by matrix: payment fraud | top by expected loss: ransomware on records
matrix ties at top score: 2

3.4 Risks to the study

Amara lists risks to her study: assessors may not complete calibration training; real incidents during the study may change attitudes; the reference panel may itself be biased; and Corvane may reorganise. For each she gives a mitigation, such as a short training module, recording incidents as covariates, using three independent panel members and agreeing a minimum participation level with Corvane in writing. A doctoral proposal that names its risks and mitigations is more credible to examiners and partners than one that assumes everything will go to plan.

Worked example

Amara structures a proposal with three questions, defines agreement, Brier score and decision-order outcomes and uses a six-scenario model to show matrix ties and rank reversal.

Synthetic case. Scenarios are synthetic; Cox (2008) is real.

Errors and limits

Small models illustrate, they do not prove. Reference panels can share biases.

Exercises

Synthesis

A strong proposal defines measurable outcomes and shows with a model why the question matters.

Continue studying — chapter 03 of 08

← Chapter 02 (draft) · Course chapter index · Chapter 04 (draft) →