Chapter 3 — Programming I
Lesson map
Lesson 03 published. Map, example, EX007.
flowchart TD
N0["Learning outcomes"]
N1["3.1 Why analysts program"]
N2["3.2 Types for intelligence"]
N3["3.3 Classifying values"]
N4["3.4 Explicit failure"]
N5["3.5 No hidden collection"]
N6["Readable code"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX007"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["3.1 Why analysts program"]
N2["3.2 Types for intelligence"]
N3["3.3 Classifying values"]
N4["3.4 Explicit failure"]
N5["3.5 No hidden collection"]
N6["Readable code"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX007"]
[Learning outcomes]
|
v
[3.1 Why analysts program]
|
v
[3.2 Types for intelligence]
|
v
[3.3 Classifying values]
|
v
[3.4 Explicit failure]
|
v
[3.5 No hidden collection]
|
v
[Readable code]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX007]
Learning outcomes
You will (1) write Python functions that clean and classify indicators; (2) use strings, lists, sets and dictionaries; (3) handle bad input explicitly; (4) keep scripts free of network collection.
3.1 Why analysts program
Analysts receive indicators in emails, PDFs and spreadsheets. Small scripts extract, clean and compare them faster and more accurately than copying by hand. Programming here is a reading and checking tool, not a collection tool.
Amara writes her first script to read indicators from a spreadsheet shared by HSSC, remove duplicates and compare them with a list exported from the email gateway. Done by hand, the comparison took an hour and missed two matches; the script takes seconds and gives the same answer each time it runs.
3.2 Types for intelligence
A string holds one indicator. A list keeps order, such as events in time. A set removes duplicates and supports intersection — which indicators appear in both our logs and the feed? A dictionary maps an indicator to its metadata: source, TLP, first seen.
Choosing the right type makes the question easy to answer. Amara's script stores feed indicators in a set, so the overlap with HCHT's logs is a single intersection, and keeps metadata in a dictionary, so each match comes with its source and TLP label. Without the label, she could not tell which matches may be shared with other teams.
3.3 Classifying values
A simple function decides whether a string looks like an IPv4 address, a domain or a hash. The standard library's ipaddress module validates addresses; length and character checks distinguish SHA-256 hashes (64 hexadecimal characters) from other strings.
Amara tests the function with examples of each type and with tricky cases, such as a domain that contains digits or a hash with uppercase letters. A misclassified value could send a domain to an address blocklist, where it would do nothing, while the analyst believed it was blocked.
3.4 Explicit failure
If a value fits no type, return "unknown" and keep it for review. Guessing creates false indicators that may later block legitimate services.
Amara's script writes unknown values to a separate file with the line they came from, so she can check them by hand. Many turn out to be defanged indicators, such as domains with brackets around the dots, which the script now handles explicitly. Explicit failure turned a silent error into a list of improvements and kept false indicators out of the blocklists.
3.5 No hidden collection
Scripts in this course read local files only. Code that fetches web pages, queries external services or logs into platforms is outside this chapter and requires terms, permission and supervision.
Amara asks two questions of her own work: where does each input come from, and was she allowed to use it? When a source publishes only a web page, she asks the HSSC desk for an exported file rather than adding a download step to her script. Any collection step is a separate decision for her supervisor, with its own approval and record.
Readable code
Analysts share scripts with colleagues who must trust the output. Clear function names, short docstrings and a small test with known inputs make a script reviewable. A test such as feeding one valid hash, one truncated hash and one domain shows reviewers exactly how edge cases behave before the script touches real data.
Worked example
Amara receives 40 values pasted from a PDF. Her classifier labels 18 domains, 12 IPv4 addresses, 7 SHA-256 hashes and 3 unknowns — two were truncated hashes and one a domain with a stray bracket from defanging. She fixes the bracket, reports the truncated hashes back to the sender and stores 38.
Synthetic case. The PDF values are invented.
Errors and limits
Pattern checks confirm shape, not maliciousness. A valid-looking domain can still be a typo.
Exercises
- EX007 — Pick a type. Which Python type finds indicators present in both the feed and HCHT's logs, and how? Rubric: 3 = set intersection (&); 2 = set only; 1 = list loop without dedupe; 0 = none.
- EX008 — Hash shape. What shape does a SHA-256 hash have as text? Rubric: 3 = 64 hexadecimal characters; 2 = 64 characters; 1 = 32; 0 = none.
- EX009 — Truncated value. What should Amara do with the two truncated hashes? Rubric: 3 = mark unknown, do not store as indicators, report to sender; 2 = discard; 1 = pad and store; 0 = none.
Synthesis
Use the right data type, validate shape, fail explicitly and keep scripts local.
Continue studying — chapter 03 of 40
← Chapter 02 (draft) · Course chapter index · Chapter 04 (draft) →