Chapter 4 — Programming I | Engineering Threat intelligence and OSINT

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 4 — Programming I

Lesson map

Lesson 04 published. Map, example, EX010.

flowchart TD
  N0["Learning outcomes"]
  N1["4.1 Records as dictionaries"]
  N2["4.2 Pure functions first"]
  N3["4.3 Defanged text"]
  N4["4.4 Tests"]
  N5["4.5 What not to program"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX010"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["4.1 Records as dictionaries"]
  N2["4.2 Pure functions first"]
  N3["4.3 Defanged text"]
  N4["4.4 Tests"]
  N5["4.5 What not to program"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX010"]

[Learning outcomes]
    |
    v
[4.1 Records as dictionaries]
    |
    v
[4.2 Pure functions first]
    |
    v
[4.3 Defanged text]
    |
    v
[4.4 Tests]
    |
    v
[4.5 What not to program]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX010]

Learning outcomes

You will (1) write small, pure functions that transform intelligence records; (2) use standard-library types for strings, sets and dictionaries; (3) test functions against known inputs; (4) refuse to write collection code that violates terms or privacy.

4.1 Records as dictionaries

A parsed indicator is naturally a dictionary: {"type": "domain", "value": "c2.example", "source": "partner-A", "tlp": "TLP:GREEN"}. Functions that take one record and return a new one are easy to test and compose.

The dictionary keeps each field named, so code reads clearly and missing fields can be checked. Kofi's functions never change the record they receive; they return a new one, which means an earlier version is still available if a later step goes wrong. A record that lacks a source or a TLP label is rejected at this stage rather than stored without them.

4.2 Pure functions first

A pure function has no side effects: given the same input it returns the same output and writes nothing. Normalising a domain to lower case is pure. Writing to the database is not. Keep the pure parts separate so most of the code can be tested without infrastructure.

Kofi's parser follows that split. The functions that clean and validate a record are pure and have dozens of quick tests. A thin outer layer reads from the feed and writes to the store, and that layer is tested less often against a test database. When a bug appears, the pure tests usually locate it in seconds.

4.3 Defanged text

Reports often "defang" indicators — hxxp, [.] — so that nobody clicks them. A parser must refang them only inside the platform, never in outputs that humans will paste into browsers. Lantern stores the normal form and renders the defanged form in reports.

The two forms must never be confused. If a report shows a live link and an analyst clicks it, the click may reach attacker infrastructure, which the book's ethical boundary forbids. Kofi's rendering function defangs every domain and address, and a test fails if any report output contains a working scheme.

4.4 Tests

For each function, write three known-answer cases: a normal value, an edge case, and malformed input. Malformed input should produce an explicit error or a rejected flag, never a guessed value.

For a function that normalises a domain, the normal case might be a mixed-case name that should come back in lower case. The edge case could be a name with a trailing dot, and the malformed case a string with a space inside it. Writing these three tests before the function itself often reveals what the function should do when the input is strange, which is where real feeds cause trouble.

4.5 What not to program

Students sometimes ask how to scrape a social platform for threat actors' personal details. Lantern's code collects only from sources whose terms permit it and only the fields needed. Automated collection that evades rate limits, logins or terms is out of scope.

The rule protects people as well as BEN. Personal details of a suspected actor may belong to someone uninvolved, and gathering them can breach privacy law and platform terms. Lantern's sources are listed with the terms that allow their use, and adding a new source needs the CTI lead's approval before any code is written. Kofi's code reviews check that list.

Worked example

Kofi writes refang(s) replacing hxxp and [.]. His tests: "hxxp://a[.]example" → "http://a.example"; "a.example" unchanged; "" → rejected. A colleague's version returned an empty string for empty input; Kofi's rejects it, so empty values never become stored indicators.

Synthetic case. Kofi's function and tests are invented.

Errors and limits

Copy-pasted parsing snippets can contain network calls; read them. Refanged strings in reports invite accidental clicks.

Exercises

EX010 — Purity check. Is a function that normalises case and also logs to a file pure? Why? Rubric: 3 = no; file write is a side effect; 2 = no; 1 = yes; 0 = none.
EX011 — Refang rule. Where may refanged indicators appear, and where must they stay defanged? Rubric: 3 = refanged inside platform processing/storage; defanged in human-facing reports; 2 = one half; 1 = everywhere refanged; 0 = none.
EX012 — Empty input. What should refang("") return under Kofi's design, and why? Rubric: 3 = rejected/error so empty values never stored; 2 = error only; 1 = empty string; 0 = none.

Synthesis

Small pure functions, explicit rejection, defanged outputs and collection code that respects terms.

Continue studying — chapter 04 of 50

← Chapter 03 (draft) · Course chapter index · Chapter 05 (draft) →