Chapter 12 — Threat intelligence and OSINT: data flows
Lesson map
Lesson 12 published. Map, example, EX034.
flowchart TD
N0["Learning outcomes"]
N1["12.1 The flow at CMS"]
N2["Lab — normalise and deduplicate"]
N3["12.2 Reading the result"]
N4["12.3 Why keep provenance"]
N5["12.4 Loss points"]
N6["12.5 Documentation"]
N7["12.6 Normalisation rules"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX034"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["12.1 The flow at CMS"]
N2["Lab — normalise and deduplicate"]
N3["12.2 Reading the result"]
N4["12.3 Why keep provenance"]
N5["12.4 Loss points"]
N6["12.5 Documentation"]
N7["12.6 Normalisation rules"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX034"]
[Learning outcomes]
|
v
[12.1 The flow at CMS]
|
v
[Lab — normalise and deduplicate]
|
v
[12.2 Reading the result]
|
v
[12.3 Why keep provenance]
|
v
[12.4 Loss points]
|
v
[12.5 Documentation]
|
v
[12.6 Normalisation rules]
|
v
[Worked example] --> [Exercise EX034]
Learning outcomes
You will (1) trace intelligence from source to enforcement; (2) normalise and deduplicate indicators while keeping provenance; (3) identify points where data is lost or altered; (4) document the flow.
12.1 The flow at CMS
MCX TAXII collection and email, a vendor list and public advisories arrive; Lena's scripts normalise and deduplicate them; the indicator store holds results; exports feed the DNS filter, proxy and endpoint tool; feedback returns via tickets.
Each stage in this flow has a clear owner and a clear output. Lena owns the scripts and the store, network and endpoint teams own the tools that receive exports, and tickets carry feedback such as a false positive back to her. Seeing the whole path helps her find where an indicator went missing, as the worked example below shows.
Lab — normalise and deduplicate
"""Lab (ch 12): normalise and deduplicate indicators from three synthetic sources, keeping provenance."""
rows = [("MCX", "Pay-Portal.example."), ("vendor", "pay-portal.example"), ("public", "198.51.100.23"),
("MCX", "198.51.100.23"), ("vendor", "update-svc.example"), ("public", "PAY-PORTAL.EXAMPLE")]
seen = {}
for src, v in rows:
k = v.strip().lower().rstrip(".")
seen.setdefault(k, set()).add(src)
print("raw", len(rows), "unique", len(seen))
for k in sorted(seen):
print(k, sorted(seen[k]))
Run with python3 lab_dedupe.py. Verified output:
raw 6 unique 3
198.51.100.23 ['MCX', 'public']
pay-portal.example ['MCX', 'public', 'vendor']
update-svc.example ['vendor']
12.2 Reading the result
Six raw rows collapse to three unique indicators after lower-casing and removing trailing dots. pay-portal.example was reported by MCX, the vendor and a public source; 198.51.100.23 by MCX and a public source; update-svc.example only by the vendor. The script keeps every source for each value.
Lena checks the counts by hand: three spellings of pay-portal.example, two rows for 198.51.100.23 and one for update-svc.example make six rows and three values. The normalisation is the line v.strip().lower().rstrip("."), which removes surrounding spaces, capital letters and a trailing dot.
12.3 Why keep provenance
When the vendor later withdraws update-svc.example as a false positive, Lena can remove it confidently because no other source reported it. If she had discarded sources during deduplication, she could not tell.
Removing it is safe because the store shows the vendor as the only source. If MCX had also listed update-svc.example, Lena would keep it and ask MCX whether they agreed with the vendor. Provenance therefore protects both directions: it prevents keeping a withdrawn indicator and prevents deleting one that still has independent support.
12.4 Loss points
Data is lost when spreadsheets reformat values, when exports truncate long fields, or when TLP labels are not carried into enforcement tools. Lena checks each hand-off.
Each loss point is silent: a spreadsheet that turns an address into a number, an export that cuts a long domain, or a block that loses its TLP label shows no error. Lena compares counts before and after each hand-off, which is how the shortened domains in the worked example were found. Each check is recorded on the flow diagram with its date, so a later difference can be compared with the last good count.
12.5 Documentation
A one-page diagram shows each source, script, store and export with owners and schedules. New staff learn the flow in minutes.
The diagram also records limits found at each step, such as the DNS filter's entry length. Owners and schedules matter when something breaks, because a new colleague can see whom to ask and when the next export should run. A one-page format keeps the diagram current, since it takes little effort to update. Lena updates the diagram whenever a source, script or export changes, and dates each update so readers know how current it is.
12.6 Normalisation rules
Rules are written down: lower-case domains, strip trailing dots, keep addresses as written after validation, never alter hashes except to lower-case them. Unwritten rules drift between scripts.
Hashes are only lower-cased because changing anything else would make them refer to a different file. Addresses are kept as written after validation because the ipaddress module has already confirmed they are well formed. Writing the rules down lets Lena's deduplication script and any colleague's script produce the same values. Each rule is written beside an example value, so a reader can see exactly what the rule changes and what it leaves untouched.
Worked example
The DNS filter shows fewer entries than the store. Lena traces the export: the filter accepts at most 64 characters per entry, and two long lookalike domains were silently cut. She shortens nothing; instead she sends those two to the proxy, which accepts them, and notes the limit on the diagram.
Synthetic case. The limits and domains are invented.
Errors and limits
Normalisation can merge values that are actually different, such as internationalised domains; review edge cases.
Exercises
- EX034 — Lab counts. How many raw and unique indicators did the lab report, and who reported pay-portal.example? Rubric: 3 = 6 raw, 3 unique; MCX, public, vendor; 2 = counts only; 1 = wrong; 0 = none.
- EX035 — Withdrawal confidence. Why could Lena remove update-svc.example confidently? Rubric: 3 = only vendor reported it; provenance kept; 2 = partial; 1 = guessed; 0 = none.
- EX036 — Silent truncation. What did Lena do about the filter's length limit? Rubric: 3 = routed long domains to proxy; documented limit; 2 = one; 1 = truncated; 0 = none.
Synthesis
Trace the flow end to end, deduplicate with provenance, find loss points and document the path.
Continue studying — chapter 12 of 30
← Chapter 11 (draft) · Course chapter index · Chapter 13 (draft) →