◇ OpenCyber Academy
Technical · Chapter 8 · English

Chapter 8 — Threat intelligence and OSINT: failure modes

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 8 — Threat intelligence and OSINT: failure modes

Lesson map

Lesson 08 published. Map, example, EX029.

flowchart TD
  N0["Learning outcomes"]
  N1["8.1 Stages"]
  N2["8.2 False positives"]
  N3["8.3 False negatives"]
  N4["8.4 Label failures"]
  N5["8.5 Overload"]
  N6["8.6 Learning"]
  N7["8.7 Recording failures"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX029"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["8.1 Stages"]
  N2["8.2 False positives"]
  N3["8.3 False negatives"]
  N4["8.4 Label failures"]
  N5["8.5 Overload"]
  N6["8.6 Learning"]
  N7["8.7 Recording failures"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX029"]

[Learning outcomes]
    |
    v
[8.1 Stages]
    |
    v
[8.2 False positives]
    |
    v
[8.3 False negatives]
    |
    v
[8.4 Label failures]
    |
    v
[8.5 Overload]
    |
    v
[8.6 Learning]
    |
    v
[8.7 Recording failures]
    |
    v
[Worked example] --> [Exercise EX029]

Learning outcomes

You will (1) recognise how CTI support fails; (2) classify failures by stage; (3) spot false positives and their causes; (4) recommend fixes.

8.1 Stages

Failures happen in collection (missed alert), processing (parsing error), analysis (wrong relevance call), dissemination (wrong recipient) and action (owner never acted).

Naming the stage tells the team where to look for the fix. A missed alert points to how sources are monitored, while a wrong recipient points to dissemination rules. At Ashgrove, the unread ESX alerts during the supervisor's illness were a collection failure, fixed by the shared mailbox with two readers. Recording the stage in the failure log makes patterns across a term visible, such as repeated dissemination errors.

8.2 False positives

Blocking a legitimate site because an indicator was stale or shared. Typical causes: old cloud addresses, popular file-sharing domains in feeds, typing errors.

The revision website is the Ashgrove example: a six-month-old address that had since been reassigned to a legitimate service. Students noticed the block before the team did, which shows how false positives reach users first. Expiry dates and the allow list are the two controls that reduce them, and both are checked before each export. Popular file-sharing domains in vendor feeds are a second common source, as the allow-list catch showed.

8.3 False negatives

Missing a real match because of normalisation differences (trailing dot, capital letters), missing logs or retention expiry.

A domain written with a trailing dot in a feed will not match the same domain in DNS logs unless both are normalised. Capital letters cause the same matching problem in some college tools. Tomás's validation script lowercases names and strips trailing dots before matching, and the failure log records any new normalisation fault. Missing logs and retention expiry produce false negatives that look exactly like real negatives, so the note must name both.

8.4 Label failures

Sharing AMBER material with ESX at large. Causes: missing label in source, copy-paste stripping labels, unclear rules.

A label failure can harm the college's standing in ESX, because members share AMBER material on trust. Copy-paste from one tool to another is the commonest cause, since labels often sit in metadata rather than text. The pre-share checklist asks Tomás to confirm the label on the source before anything leaves Ashgrove. Unclear rules are the third cause, which is why the safety sheet spells out what each TLP label allows. A review that finds a label failure checks the source item, the tool used and the rule the person followed.

8.5 Overload

Too many low-value alerts mean owners stop reading. Cause: no triage before dissemination.

When owners receive alerts that rarely apply to their systems, they begin to skip them, and a relevant one is then missed. Triage using the four questions removes items that are not relevant or not actionable before they reach owners. The weekly triage with his supervisor keeps that filter consistent between them. Grouping several minor items into the Monday list or the monthly summary also reduces the number of separate messages owners receive. Owners then read each urgent alert because they know it is rare.

8.6 Learning

Each failure gets a short blameless review: what happened, why, what changed. Patterns guide improvement.

A blameless review asks what in the process allowed the failure, not who is at fault. In the revision-site case, the review found that address entries kept too long, and the expiry was shortened to 30 days. People report their own mistakes faster when they know the review looks at the process. The review is written up in the failure log within a week, while the people involved still remember the details. Changes that come out of it are given an owner and a date.

8.7 Recording failures

A shared failure log with date, stage, cause and fix shows recurring issues. Three label failures in a term indicate a process problem, not three careless people.

The failure log at Ashgrove is a shared spreadsheet that the supervisor reviews each month. Recurring causes, such as stale addresses or stripped labels, become process changes rather than reminders. Each entry links to its ticket, so the original evidence can be reread if the pattern later needs a closer look. The term's count of each cause is reported to the supervisor at the end of term.

8.8 Bias

Assuming every issue is an attack leads to over-blocking; assuming none is leads to missing real ones. Check evidence each time.

Tomás guards against both errors by writing down what the evidence shows before deciding. In the revision-site case, checking the feed's age showed a stale address, not an attack. In the spraying case, two successful logins from ESX addresses were treated as real compromise and acted on at once. Writing the evidence down first also lets the supervisor check his reasoning when she reviews the case.

Dissemination failures

An advisory sent to a general IT mailbox that nobody owns will wait for days. Each item type has a named recipient, and the sender asks for acknowledgement so silence is noticed. When no acknowledgement arrives within the agreed time, the item is escalated to the supervisor rather than left waiting.

Worked example

Students complain that a revision website is blocked. Tomás finds the address came from a feed six months old; the site moved to that cloud address last month. He requests removal, the network team unblocks it within an hour, and the expiry rule for addresses is shortened to 30 days.

Synthetic case. The site and feed are invented.

Errors and limits

Fixing one failure can create another; test changes.

Exercises

Synthesis

Classify failures by stage, trace causes, review without blame and fix the process.

Continue studying — chapter 08 of 18

← Chapter 07 (draft) · Course chapter index · Chapter 09 (draft) →