Chapter 13 — Threat intelligence and OSINT: troubleshooting
Lesson map
Lesson 13 published. Map, example, EX049.
flowchart TD
N0["Learning outcomes"]
N1["13.1 Common problems"]
N2["13.2 Step by step"]
N3["13.3 Tool logs"]
N4["13.4 Communicating"]
N5["13.5 Prevention"]
N6["13.6 Keep changes small"]
N7["13.7 Escalation"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX049"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["13.1 Common problems"]
N2["13.2 Step by step"]
N3["13.3 Tool logs"]
N4["13.4 Communicating"]
N5["13.5 Prevention"]
N6["13.6 Keep changes small"]
N7["13.7 Escalation"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX049"]
[Learning outcomes]
|
v
[13.1 Common problems]
|
v
[13.2 Step by step]
|
v
[13.3 Tool logs]
|
v
[13.4 Communicating]
|
v
[13.5 Prevention]
|
v
[13.6 Keep changes small]
|
v
[13.7 Escalation]
|
v
[Worked example] --> [Exercise EX049]
Learning outcomes
You will (1) diagnose problems in imports, matching and exports; (2) use logs of your own tools; (3) communicate pipeline problems; (4) add checks to prevent repeats.
13.1 Common problems
TAXII import fails; a CSV column renamed; matches drop to zero; a block list fails to load; a label value appears that scripts do not recognise.
Each of these problems appears first as a change in counts or an error in a log. A renamed CSV column can make a vendor list load as empty without any error at all. An unknown label value, such as a misspelt TLP marking, must stop the item rather than default to sharing. Each is listed in the fault log with its usual symptom.
13.2 Step by step
Check input arrived, parsing succeeded, storage updated, export produced, control loaded. At each step compare counts with yesterday.
Following the pipeline in order finds the first step where counts differ from the day before. In the Tuesday case, DNS logs had arrived but the matcher read none, which placed the fault in parsing. Checking all five steps each time avoids blaming a control when the fault lies earlier. Comparing with yesterday works only if counts are logged each day, which is why the scripts record them. Counts from the same weekday are better references during term, because volumes follow the timetable.
13.3 Tool logs
Scripts write their own logs: start time, inputs, counts, errors. Without them, troubleshooting is guesswork.
Tomás's scripts write one line per run with the start time, input file, number of records read, number matched and any errors. The zero count that revealed Tuesday's fault came from these run logs. They are kept in the team repository's log folder, where the supervisor can read them when Tomás is away. Errors are written with the line number or record that caused them. Logs older than the DNS retention period are archived, since no rerun could use the data anyway.
13.4 Communicating
If protection lapsed, tell owners plainly: what stopped, since when, what still works, when a fix is expected.
Owners need to know whether they are exposed and for how long. A message that says DNS matching stopped on Tuesday morning, while email blocking still works, lets them judge the risk. Giving an expected fix time, even a rough one, saves them chasing for updates while the fault is being repaired. Messages about lapses go to the owners' shared mailboxes, never to one person. A second message confirms when protection is restored, with the time the rerun finished.
13.5 Prevention
After each fix, add a check: alert if imports are zero; alert on unknown labels; compare exported and loaded counts.
Each check turns a fault that once went unnoticed into an alert. The zero-parse warning added after the Tuesday case would show the same problem the first day it happened. Comparing exported and loaded counts catches lists that failed to load into the DNS filter or gateway. Each new check is listed in the fault log entry that prompted it. Tomás tests each new check on synthetic data before it goes live, to confirm it fires when it should.
13.6 Keep changes small
Change one thing at a time while troubleshooting; otherwise you cannot tell which change helped.
In the Tuesday case, Tomás changed only the parser to read fields by name, then reran the data. If he had also changed the matching rules, he could not have said which change restored the three matches. Each change is committed separately in the repository with a short note. Small changes also make rollback simple, because only one commit has to be reverted. After each change, Tomás reruns the sample file of log lines to confirm nothing else broke.
13.7 Escalation
When the problem lies in another team's system, give them evidence: the step where counts stop, timestamps and error messages.
The resolver belongs to the network team, so a format change there is theirs to explain and to announce in future. Showing them the step where counts stopped, with times and the parser's error lines, saves them repeating the diagnosis. Tomás asks to be told before future updates that change log formats. The evidence is attached to a ticket in the other team's queue, not sent in a private message. Tomás follows up if the ticket has no reply within the agreed service time.
13.8 Recording
A fault log with symptom, cause, fix and prevention builds team memory.
The fault log uses the same fields for every entry, so similar faults can be found quickly. The Tuesday entry records the resolver update as cause, parsing by name as fix and the zero-count warning as prevention. New team members read the log to learn how the pipeline usually breaks. Entries are written as soon as the fix is confirmed, while details are fresh. Each entry links to the commit that fixed the fault and the ticket that reported it.
13.9 Calm under pressure
Pipeline faults during an incident feel urgent. Follow the same steps; skipping them usually costs more time.
During an incident, the temptation is to change several things at once to restore matching quickly. Following the steps in order usually finds the fault faster than guessing. Tomás keeps notes as he works, so the incident lead can see what has been checked and what remains. Tomás tells the incident lead how long the next step will take, so they can plan around the gap. If the fault cannot be fixed quickly, he says which fallback from the dependency plan applies.
Test data for the parser
Keep a small file of sample log lines in each known format, including the new one. Running the parser against it after every change proves old and new formats both still work.
Worked example
Matches fall to zero on Tuesday. Counts show DNS logs arrived but the matcher read none. The log format gained a new first column after a resolver update, shifting fields. Tomás updates the parser to read fields by name, reruns Tuesday's data — finding three matches — and adds a check that warns when a day's parse count is zero.
Synthetic case. The update is invented.
Errors and limits
Some faults are intermittent; keep logs long enough to see patterns.
Exercises
- EX049 — Zero matches. What caused the zero matches, and what was the lasting fix? Rubric: 3 = new first column shifted fields; parse by name + zero-count check; 2 = one; 1 = vague; 0 = none.
- EX050 — One change rule. Why change one thing at a time? Rubric: 3 = know which change worked; 2 = partial; 1 = faster to change all; 0 = none.
- EX051 — Own logs. What should scripts record in their own logs? Rubric: 3 = start time, inputs, counts, errors; 2 = two; 1 = one; 0 = none.
- EX052 — Rerun. Why rerun Tuesday's data after the fix? Rubric: 3 = find missed matches during the fault; 2 = partial; 1 = unnecessary; 0 = none.
Synthesis
Troubleshoot stage by stage with counts and tool logs, communicate lapses, rerun missed data and add checks.
Continue studying — chapter 13 of 18
← Chapter 12 (draft) · Course chapter index · Chapter 14 (draft) →