Chapter 9 — Threat intelligence and OSINT: leadership and assurance | Postgraduate Threat intelligence and OSINT

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 9 — Threat intelligence and OSINT: leadership and assurance

Lesson map

Lesson 09 published. Map, example, EX041.

flowchart TD
  N0["Learning outcomes"]
  N1["9.1 Leading analysts"]
  N2["9.2 Quality assurance"]
  N3["9.3 Calibration over time"]
  N4["9.4 Board reporting"]
  N5["9.5 Sustainability"]
  N6["9.6 External assurance"]
  N7["Hiring and development"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX041"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["9.1 Leading analysts"]
  N2["9.2 Quality assurance"]
  N3["9.3 Calibration over time"]
  N4["9.4 Board reporting"]
  N5["9.5 Sustainability"]
  N6["9.6 External assurance"]
  N7["Hiring and development"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX041"]

[Learning outcomes]
    |
    v
[9.1 Leading analysts]
    |
    v
[9.2 Quality assurance]
    |
    v
[9.3 Calibration over time]
    |
    v
[9.4 Board reporting]
    |
    v
[9.5 Sustainability]
    |
    v
[9.6 External assurance]
    |
    v
[Hiring and development]
    |
    v
[Worked example] --> [Exercise EX041]

Learning outcomes

You will (1) lead a CTI team; (2) assure analytic quality with review processes; (3) report to boards; (4) build a sustainable programme.

9.1 Leading analysts

Analysts need clear requirements, time for depth, and protection from constant interruption. A leader shields the team from ad hoc requests by routing them through the requirements process. At NLG, requests from executives arrived by email and in corridors, pulling analysts away from planned work. Rafael routes every request through the requirements process, where it is scoped and prioritised. Analysts get protected time for deeper work, and requesters get a clear answer about when to expect results.

Urgent requests still reach the team quickly, but they are logged so their volume is visible.

9.2 Quality assurance

Use ICD 203 standards as a review checklist: sourcing described, uncertainty expressed, assumptions separated from judgments, alternatives considered, relevant and logical. Every assessment above routine level gets a second reader. Track how often reviews change conclusions. After three months of Rafael's review log, second readers changed the confidence level in 30% of assessments and the main judgment in 5%. He uses these figures to show the board that review adds value, and to find where analysts most often need help.

The checklist is kept short so second readers use it every time.

9.3 Calibration over time

Record probability judgments and later outcomes; compute how often "likely" events happened. Calibration feedback improves judgment more than training slides. Rafael starts a simple record of every probability judgment his team makes, such as likely or unlikely, with the date and the outcome when it becomes known. After enough judgments, the team can see whether its likely really happens more often than its unlikely. He discusses results in team meetings without blaming individuals, since the aim is better-calibrated words for the whole team.

9.4 Board reporting

Boards need a short view: what threats matter, how NLG's exposure is changing, what decisions intelligence informed, and what is uncertain. Avoid dramatic imagery and lists of actors. At NLG, Rafael's board report covered the threats most relevant to logistics, changes in exposure such as supplier access, and two decisions intelligence informed: segmentation funding and supplier access restrictions. It also stated what remained uncertain, such as the leading explanation for the data theft. The board could see the function's value from those decisions without reading technical detail.

9.5 Sustainability

Document processes, cross-train, keep a budget line, and show value with outcome measures. Programmes built around one person collapse when that person leaves. Rafael documents NLG's processes, trains the analyst who will succeed him and keeps a budget line for FPSC membership and the platform. When he was promoted, the handover worked because these were in place: six months later, the decision-record practice continued without him.

He also keeps the source catalogue and platform documentation current, so a new analyst can start without needing him. The cross-training plan pairs each process with a second person who has carried it out at least once under supervision.

9.6 External assurance

Occasional peer review by another FPSC member's CTI lead — under a mutual non-disclosure agreement — gives an outside view of method quality. A CTI lead from another FPSC member reviews a sample of NLG assessments against ICD 203 standards and gives written feedback. The mutual non-disclosure agreement protects both organisations' material, including anything shared at AMBER. Rafael records the findings and actions, and offers the same review in return.

The review covers method quality, not NLG's internal incidents, so it can be arranged without sharing sensitive operational detail. Rafael schedules it once a year, and the first review recommended clearer statements of assumptions.

Hiring and development

Hire for curiosity, writing and judgment as much as for tool skills; tools change faster than habits of mind. Give each analyst a development plan with one analytic skill, one technical skill and one domain topic per year, such as logistics operations or customs systems. Pairing junior analysts with reviewers on real products teaches faster than courses alone, and rotating analysts briefly through incident response keeps their work grounded in operational reality.

Worked example

Rafael introduces a review log. After three months, second readers changed the confidence level in 30% of assessments and the main judgment in 5%. He presents these figures to the board as evidence that review adds value, alongside two decisions intelligence informed: segmentation funding and supplier access restrictions.

Synthetic case. The figures are invented.

Errors and limits

Quality checklists can become box-ticking; rotate reviewers and discuss real cases.

Exercises

EX041 — Request shielding. How does a leader protect analysts from ad hoc requests? Rubric: 3 = route through requirements process/prioritise; 2 = partial; 1 = accept all; 0 = none.
EX042 — Review checklist. Name four ICD 203-based review checks. Rubric: 3 = four valid standards; 2 = three; 1 = one; 0 = none.
EX043 — Calibration method. Describe how to measure analysts' calibration. Rubric: 3 = record judgments with probabilities, compare outcome frequencies per band; 2 = partial; 1 = survey; 0 = none.
EX044 — Board content. What four things should a board CTI report cover? Rubric: 3 = threats, exposure change, decisions informed, uncertainty; 2 = three; 1 = actor list; 0 = none.
EX045 — Review value. Interpret the review-log figures. Rubric: 3 = review often adjusts confidence, rarely main judgment; adds value; 2 = partial; 1 = wasteful; 0 = none.

Synthesis

Lead through requirements, assure quality with standards-based review and calibration, report clearly to boards and build for continuity.

Continue studying — chapter 09 of 12

← Chapter 08 (draft) · Course chapter index · Chapter 10 (draft) →