◇ OpenCyber Academy
Professional · Chapter 10 · English

Chapter 10 — Threat intelligence and OSINT: quality and troubleshooting

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 10 — Threat intelligence and OSINT: quality and troubleshooting

Lesson map

Lesson 10 published. Map, example, EX046.

flowchart TD
  N0["Learning outcomes"]
  N1["10.1 Quality dimensions"]
  N2["10.2 Common faults"]
  N3["10.3 Troubleshooting"]
  N4["10.4 Feedback"]
  N5["10.5 Improvements"]
  N6["10.6 Accuracy tracking"]
  N7["10.7 Source review"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX046"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["10.1 Quality dimensions"]
  N2["10.2 Common faults"]
  N3["10.3 Troubleshooting"]
  N4["10.4 Feedback"]
  N5["10.5 Improvements"]
  N6["10.6 Accuracy tracking"]
  N7["10.7 Source review"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX046"]

[Learning outcomes]
    |
    v
[10.1 Quality dimensions]
    |
    v
[10.2 Common faults]
    |
    v
[10.3 Troubleshooting]
    |
    v
[10.4 Feedback]
    |
    v
[10.5 Improvements]
    |
    v
[10.6 Accuracy tracking]
    |
    v
[10.7 Source review]
    |
    v
[Worked example] --> [Exercise EX046]

Learning outcomes

You will (1) measure the quality of CTI products; (2) detect and fix common faults; (3) collect consumer feedback; (4) improve continuously.

10.1 Quality dimensions

Relevance, timeliness, accuracy, clarity and actionability. Each can be checked: was the item about BIB systems? delivered before it mattered? confirmed or withdrawn later? understood? acted on? Each dimension gets a simple test from records Priya already keeps. Relevance comes from the ticket's requirement field; timeliness from arrival and action dates; accuracy from whether a judgment was later confirmed or withdrawn. Clarity shows in follow-up questions from readers, and actionability in whether a request was carried out. Together the five tests show at which point a particular product failed its reader.

10.2 Common faults

Stale indicators causing false blocks; missed items because email alerts went to spam; wrong labels; notes without clear actions. Each of these four faults has a typical cause in the way BIB works. Stale indicators come from blocks that were never expired; alerts in spam come from a mail filter treating FSSG's bulk messages as marketing. Wrong labels slip in when items are copied quickly, and notes without actions usually mean the analysis stopped at describing the threat rather than deciding what BIB should do.

10.3 Troubleshooting

Trace the item's path: received, triaged, analysed, sent, acted. Find where it stopped. The ticket fields recorded for every item make tracing a lost item possible. If an expected FSSG alert never reached triage, Priya checks whether it arrived and then looks in mail filter quarantine and spam folders. If it reached analysis but nobody acted, the request may have lacked an owner or deadline. Fixing the stage where it stopped prevents the same loss next time. The trace is recorded in the ticket, so the same break in the path can be checked first if it recurs.

10.4 Feedback

Quarterly two-question forms; quick chats with the help desk. "What helped? What was noise?" The help-desk chats are often more useful than the forms because they are quick and specific. Those conversations revealed that the Wednesday scam update arrived after Monday's peak of suspicious calls. 'What was noise?' matters as much as 'What helped?', because removing a useless product frees part of Priya's single day for work that someone actually uses. The answers go into the quarterly requirements review alongside the forms, together with the dates of the chats they came from.

10.5 Improvements

One change per month: shorter notes, earlier scam updates, new lookalike monitoring. Changing one thing at a time lets Priya see what each change did. Moving the scam update to Monday was one such change; calls needing escalation fell slightly over the next month. Had she also shortened notes in the same month, she could not tell which change helped. Each change and its result goes into the improvement log. Waiting a month between changes is slow, but with one working day a week it is the only way to see effects.

10.6 Accuracy tracking

Record judgments with probability words; later note what happened. Over time, see if "likely" events happened more often than "unlikely" ones. The record needs only the date, the judgment, the probability word and the later outcome. With a small team, the sample stays small, so Priya treats the comparison as a rough indicator. If events she called 'likely' happened no more often than 'unlikely' ones, she would revisit how she chooses words rather than abandon the scale. The scale itself is never adjusted quietly, because older judgments must stay comparable with the newer ones being tracked.

10.7 Source review

Each quarter, review sources: items received, used, withdrawn. The quarterly review compares each source's items received with those used and those later withdrawn or corrected. A source that sends many items but few useful ones costs reading time. A vendor blog graded C may move to B if its claims keep proving accurate. Changes to source grades are recorded so later scores can be understood. A source that is withdrawn often is downgraded, and the reason is noted for anyone reading older tickets that relied on it.

10.8 False positives

When a block is wrong, remove it quickly, apologise, find the cause, adjust rules. Speed limits the harm a wrong block does to brokers, partners and BIB's reputation. The false-positive block in her portfolio review was fixed within an hour. Apologising to affected users keeps their trust in future requests, and finding the cause, such as a stale indicator or a missing check with operations, leads to a change in rules rather than simply removing the block. The wrong block and its fix are also recorded in the control register beside the original request that caused it.

10.9 Documentation

Keep a short improvement log. Each entry records the date, the change, the reason and what happened afterwards. The move of the scam update from Wednesday to Monday is a typical entry in the log. The log goes into the portfolio in Chapter 12 and shows assessors that improvements came from evidence rather than habit. It also helps the IT manager understand why the routine looks the way it does. Entries are kept short, often two or three lines, so that writing them never competes with the weekly routine itself.

Peer comparison

Twice a year Priya swaps an anonymised product with a peer at another FSSG member and each reviews the other's work against the same quality dimensions. The outside view catches habits she no longer notices, such as long sentences or vague actions, and gives her ideas from a team facing similar threats with different tools. The peer review is covered by FSSG's sharing agreement, and only TLP:GREEN or CLEAR material is exchanged.

Worked example

The help desk says scam updates arrive on Wednesday, after Monday's peak of suspicious calls. Priya moves the update to Monday morning using the previous week's data. Calls needing escalation fall slightly over the next month; she records the change and result in her improvement log.

Synthetic case. The change is invented.

Errors and limits

Small teams have small samples; treat measures as indicators.

Exercises

Synthesis

Measure quality, trace faults, gather feedback, make one improvement at a time and track accuracy.

Continue studying — chapter 10 of 12

← Chapter 09 (draft) · Course chapter index · Chapter 11 (draft) →