Chapter 5 — Proposal and pilot | Master's Threat intelligence and OSINT

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 05 published. Map, example, EX029.

flowchart TD
  N0["Learning outcomes"]
  N1["5.1 Aims and measures"]
  N2["5.2 Data dictionary"]
  N3["5.3 Pilot"]
  N4["5.4 Revising without drifting"]
  N5["5.5 Indicator formats"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX029"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["5.1 Aims and measures"]
  N2["5.2 Data dictionary"]
  N3["5.3 Pilot"]
  N4["5.4 Revising without drifting"]
  N5["5.5 Indicator formats"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX029"]

[Learning outcomes]
    |
    v
[5.1 Aims and measures]
    |
    v
[5.2 Data dictionary]
    |
    v
[5.3 Pilot]
    |
    v
[5.4 Revising without drifting]
    |
    v
[5.5 Indicator formats]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX029]

Learning outcomes

You will (1) write a dissertation proposal with measurable aims; (2) define each variable in a data dictionary; (3) run a pilot to check feasibility; (4) revise the plan from pilot evidence without changing the question.

5.1 Aims and measures

Aim 1: measure median time-to-block for indicators received via CPSG. Aim 2: compare with matched non-members. Aim 3: describe how often shared indicators were judged not worth blocking. Each aim names its variable and its analysis.

Aim 3 is easy to forget but matters for interpretation. If members ignore many shared indicators as low-value, a fast median for the blocked ones may hide the fact that most indicators never become controls. Reporting the blocked, monitored and ignored proportions beside the median keeps the result honest.

5.2 Data dictionary

t_pub: timestamp the indicator was first published to the TAXII collection (UTC). t_block: timestamp a block rule became active in the member's control. ttb_h: hours between them. ioc_type: domain, ip, hash or url. decision: blocked, monitored, ignored. Every field has a type, unit and allowed values.

The dictionary also decides how a missing value is handled: an indicator that was ignored has no t_block, so its ttb_h is empty rather than zero, and it is counted under Aim 3, not in the median. Fixing that rule before data arrive prevents ignored indicators being treated as instant blocks.

5.3 Pilot

A pilot with one member and two weeks of synthetic-format logs tests whether t_block is recorded at all. Many organisations log rule edits only in change tickets with day-level precision, which would make hour-level timings impossible.

Lucía's pilot found exactly this split: the first member's firewall recorded rule activation to the second, while the second member's ticketing tool held dates only. Finding it in the pilot, with two weeks of logs, cost little; finding it after the main collection would have forced a redesign.

5.4 Revising without drifting

If the pilot shows day-level precision only, Lucía can change the unit to days — a measurement revision — but should not change the question to something easier, such as "do members like the feed?" That would be scope drift.

Lucía's revision kept both units: hours for members with precise logs, and days for the others as a separate stratum, with the reduced precision noted as a limitation. The question, time from publication to block, did not change; only how precisely some members could answer it.

5.5 Indicator formats

CPSG publishes STIX 2.1 indicator objects. The pilot confirms that created and valid_from are populated, giving a trustworthy t_pub. Objects missing required properties are logged as data-quality findings for the community.

The data-quality findings are recorded by publishing member class, not by member, following the board's aggregation rule. An indicator without valid_from still counts for Aim 3 if a member acted on it, but it cannot contribute a publication time to the median. The pilot member's two weeks of logs gave Lucía a first count of such findings before the main export arrived.

Worked example

The pilot member's firewall records rule activation to the second, but the second member uses a ticketing tool with dates only. Lucía keeps hours for members with precise logs, records days for the others as a separate stratum, and notes the reduced precision as a limitation.

Synthetic case. Pilot members and logging tools are invented.

Errors and limits

A pilot is not a small version of the result; do not report its medians as findings. Changing the question after a pilot requires supervisor approval and a new ethics check.

Exercises

EX029 — Variable entry. Write a complete data-dictionary entry for ttb_h.
Show marking criteria
Rubric: 3 = name, definition, type, unit, derivation; 2 = three of those; 1 = name only; 0 = none.
EX030 — Pilot purpose. What specific feasibility question does the two-week pilot answer?
Show marking criteria
Rubric: 3 = whether t_block is logged with usable precision; 2 = data exists generally; 1 = whether sharing works; 0 = none.
EX031 — Precision fix. How does Lucía handle a member logging dates only?
Show marking criteria
Rubric: 3 = separate day-level stratum + limitation; 2 = drop member; 1 = guess hours; 0 = none.
EX032 — Drift test. Is switching to "do members like the feed?" a revision or drift? Explain.
Show marking criteria
Rubric: 3 = drift; changes construct away from time-to-block; 2 = drift; 1 = revision; 0 = none.
EX033 — Aim three. Why is the "ignored" decision value important for Aim 3?
Show marking criteria
Rubric: 3 = captures indicators judged not worth blocking; avoids survivorship view; 2 = partial; 1 = irrelevant; 0 = none.
EX034 — STIX time. Which STIX 2.1 properties give a trustworthy publication time?
Show marking criteria
Rubric: 3 = created (and valid_from) on the indicator; 2 = one; 1 = name; 0 = none.
EX035 — Pilot numbers. May Lucía cite the pilot median in her results chapter?
Show marking criteria
Rubric: 3 = no; pilot only tests feasibility; mention in methods; 2 = no; 1 = yes; 0 = none.

Synthesis

Measurable aims, a precise dictionary, a pilot aimed at feasibility, and revisions that keep the question intact.

Continue studying — chapter 05 of 08

← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →