◇ OpenCyber Academy
Doctoral research · Chapter 3 · English

Chapter 3 — Original contribution proposal

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 3 — Original contribution proposal

Lesson map

Lesson 03 published. Map, example, EX015.

flowchart TD
  N0["Learning outcomes"]
  N1["3.1 Falsifiable questions"]
  N2["3.2 Constructs and measures"]
  N3["3.3 Design that can fail"]
  N4["3.4 Threats to validity"]
  N5["3.5 Contribution claim"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX015"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["3.1 Falsifiable questions"]
  N2["3.2 Constructs and measures"]
  N3["3.3 Design that can fail"]
  N4["3.4 Threats to validity"]
  N5["3.5 Contribution claim"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX015"]

[Learning outcomes]
    |
    v
[3.1 Falsifiable questions]
    |
    v
[3.2 Constructs and measures]
    |
    v
[3.3 Design that can fail]
    |
    v
[3.4 Threats to validity]
    |
    v
[3.5 Contribution claim]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX015]

Learning outcomes

You will (1) frame a falsifiable research question; (2) define constructs and their operational measures; (3) choose a design that can fail; (4) write a proposal section on threats to validity before data collection.

3.1 Falsifiable questions

"Confidence language matters" cannot fail. "Statements labelled 'highly likely' in the corpus are later confirmed at a rate at least 20 percentage points above statements labelled 'possible'" can fail. A doctoral contribution earns its name by risking refutation.

The 20-point threshold also gives the committee a number to argue with before the data arrive. If they think a 10-point difference would already matter for blocking decisions, the right time to change the threshold is in the proposal, recorded in the pre-registration, not after the calibration curve is drawn.

3.2 Constructs and measures

Construct: analytic confidence as communicated. Measure: the estimative phrase in the key judgment, mapped to a numeric range on a published scale, for example Kent's words of estimative probability. Construct: confirmation. Measure: whether a later authoritative source in the corpus — an indictment summary, a vendor retraction, or a joint advisory — supports or contradicts the claim within 24 months. Every mapping decision is written in a codebook.

The 24-month window is itself a design choice with a cost: claims made late in the corpus have less time to be confirmed, so they are more likely to remain unresolved. The codebook therefore records the report date, and the analysis can check whether resolvability falls for recent reports.

3.3 Design that can fail

Tomas chooses a retrospective cohort design: sample key judgments, code their stated probability, code outcomes blind to the original phrase, and compute calibration. The design fails if outcomes are unknowable for most claims, so the proposal sets a minimum of 300 resolvable judgments before analysis proceeds.

The floor is a pre-registered stopping rule, not a target. If only 180 sampled judgments turn out to be resolvable, Tomas must extend the sample or revise the design under the rule written in advance, and report the shortfall; proceeding quietly with 180 would make the high-confidence bins too thin to estimate.

3.4 Threats to validity

Internal: coders may infer the phrase from context and bias outcome coding. Construct: an "outcome" source may itself rely on the original report. External: commercial reports may differ from government ones. Statistical: small bins give unstable calibration. Writing these down now lets the committee judge the plan rather than the luck of the result.

Each threat has a planned response. Outcome coders see claims with the estimative phrase removed. The circularity check the committee requested flags outcome sources that cite the original report. Commercial and government reports are analysed separately. The main sample over-weights high-confidence judgments so that the top bins are not the smallest, which the pilot showed was the weak point.

3.5 Contribution claim

The claim is narrow: the first measured calibration estimate of estimative language in a defined synthetic-plus-licensed corpus, with an open codebook. It is not "CTI is unreliable".

Keeping the claim narrow is what makes a negative result publishable. If high-confidence statements turn out to be confirmed no more often than possible ones, the thesis still delivers a measured estimate and an open codebook that others can apply to their own corpora; a broad claim would have nowhere to retreat. The open codebook is part of the claim for that reason: it lets other teams test the same question on their own reports.

Worked example

Dr. Sato challenges the first draft question, "Are CTI vendors overconfident?" Tomas rewrites it with a measurable threshold, a defined corpus, a blind outcome-coding step and a resolvability floor. The committee approves the revised proposal with a request to add a circularity check for outcome sources.

Synthetic case. The proposal exchange is invented.

Errors and limits

Over-broad claims attract rejection; under-specified constructs cannot be replicated. A proposal is a plan, not a result.

Exercises

Synthesis

Risk refutation, define measures in a codebook, design blind coding, write validity threats first, and keep the claim exactly as wide as the evidence.

Continue studying — chapter 03 of 08

← Chapter 02 (draft) · Course chapter index · Chapter 04 (draft) →