Chapter 42 — Threat intelligence and OSINT: applied research

Lesson map

Lesson 42 published. Map, example, EX124.

flowchart TD
  N0["Learning outcomes"]
  N1["42.1 Questions from operations"]
  N2["42.2 Data and consent"]
  N3["42.3 Method"]
  N4["42.4 Humility in statistics"]
  N5["42.5 Publication"]
  N6["Practice note — reproducibility"]
  N7["Peer review inside BEN"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX124"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["42.1 Questions from operations"]
  N2["42.2 Data and consent"]
  N3["42.3 Method"]
  N4["42.4 Humility in statistics"]
  N5["42.5 Publication"]
  N6["Practice note — reproducibility"]
  N7["Peer review inside BEN"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX124"]

[Learning outcomes]
    |
    v
[42.1 Questions from operations]
    |
    v
[42.2 Data and consent]
    |
    v
[42.3 Method]
    |
    v
[42.4 Humility in statistics]
    |
    v
[42.5 Publication]
    |
    v
[Practice note — reproducibility]
    |
    v
[Peer review inside BEN]
    |
    v
[Worked example] --> [Exercise EX124]

Learning outcomes

You will (1) frame an engineering research question from Lantern's operational data; (2) design a study that uses synthetic or consented data; (3) evaluate results statistically with appropriate humility; (4) report findings for practitioners and researchers.

42.1 Questions from operations

Lantern's logs suggest questions: Does indicator half-life differ by sector partner? Does behaviour-based detection catch incidents that indicator matching misses? How much do sightings shared with partners improve their retirement decisions?

Kofi picks questions where the answer would change something BEN does. The half-life question affects how long indicators stay in blocking. The detection question affects where the team invests effort. Questions that are merely interesting, without any decision attached, wait until the useful ones are answered. Each question has an owner.

42.2 Data and consent

Internal operational data may be used for improvement under BEN's policies; partner data needs partner consent for research use and publication. Personal data is excluded. Where consent is lacking, synthetic data that reproduce statistical properties can test methods but not prove facts about the real world.

Kofi asked each partner in writing whether their data could be included in aggregate form, and two declined. Their indicators were excluded from the study, which narrowed it but kept faith with them. The study notes that exclusion.

42.3 Method

Kofi studies half-lives: for each retired indicator, did it reappear in sightings within 60 days? He fits re-observation rates by indicator type and compares with the decay model's assumptions.

The choice of sixty days follows from the existing half-lives: a window shorter than the assumed domain half-life would miss reappearances the model claims are likely. Kofi counts each indicator once, whatever the number of later sightings, to avoid a few busy indicators dominating the result. He also records indicators that were never sighted, since those carry information about decay too.

42.4 Humility in statistics

Report sample sizes and intervals. A difference between partners of two days in half-life may be noise with small samples.

Kofi computed intervals for each indicator type's re-observation rate and found that the intervals for two partners overlapped almost completely, so he did not claim a difference between them. The domain finding was clearer because that sample was much larger. His report states the number of indicators behind every rate and says plainly where the data could not separate an effect from chance.

42.5 Publication

Results are shared with partners first, in aggregate, then perhaps at a practitioner conference with BEN's approval. Raw indicators are never published.

Partners see the findings before anyone else because their data contributed and their own decay settings may need to change. A conference talk would describe methods and aggregate results, after review by BEN's communications team and the partners named in the sharing agreement. Raw indicators stay out because they could reveal which organisations were targeted, even after the campaign has ended.

Practice note — reproducibility

Code, synthetic data generators and parameter files are archived so another engineer can repeat the analysis next year.

Peer review inside BEN

Before sharing outside, a senior analyst and a data-protection reviewer read the study. The analyst checks that conclusions follow from the data; the reviewer checks that no partner or person can be identified from tables or examples.

Worked example

Re-observation data show that domain indicators are re-sighted at meaningful rates up to about 60 days, longer than the 45-day half-life assumed. Kofi proposes raising the domain half-life and documents the sample size, interval and limits.

Synthetic case. Findings are invented.

Errors and limits

Operational data reflect BEN's environment; generalise cautiously. Do not publish partner-specific statistics without consent.

Exercises

• EX124 — Research question. Write a research question Lantern data could answer.
  Show marking criteria

  Rubric: 3 = specific, measurable, uses available data; 2 = vague; 1 = unanswerable; 0 = none.
• EX125 — Synthetic limit. What can synthetic data show in this research, and what can it not?
  Show marking criteria

  Rubric: 3 = test methods/pipelines; not prove real-world facts; 2 = one half; 1 = proves all; 0 = none.
• EX126 — Publish order. In what order are results shared, and why?
  Show marking criteria

  Rubric: 3 = partners first in aggregate, then wider with approval; reciprocity/consent; 2 = order only; 1 = public first; 0 = none.

Synthesis

Ask operational questions, respect consent, test methods on synthetic data, report with humility and share with partners first.

Continue studying — chapter 42 of 50

← Chapter 41 (draft) · Course chapter index · Chapter 43 (draft) →