Chapter 8 — Programming II | Bachelor's Security operations and blue team

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 08 published. Map, example, EX022.

flowchart TD
  N0["Learning outcomes"]
  N1["8.1 From scripts to tools"]
  N2["8.2 Dates and times"]
  N3["8.3 JSON"]
  N4["8.4 Unit tests"]
  N5["8.5 Efficiency"]
  N6["8.6 Errors"]
  N7["8.7 Sharing"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX022"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["8.1 From scripts to tools"]
  N2["8.2 Dates and times"]
  N3["8.3 JSON"]
  N4["8.4 Unit tests"]
  N5["8.5 Efficiency"]
  N6["8.6 Errors"]
  N7["8.7 Sharing"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX022"]

[Learning outcomes]
    |
    v
[8.1 From scripts to tools]
    |
    v
[8.2 Dates and times]
    |
    v
[8.3 JSON]
    |
    v
[8.4 Unit tests]
    |
    v
[8.5 Efficiency]
    |
    v
[8.6 Errors]
    |
    v
[8.7 Sharing]
    |
    v
[Worked example] --> [Exercise EX022]

Learning outcomes

You will (1) structure larger analysis programs into functions and modules; (2) use the standard library for dates, CSV and JSON; (3) write unit tests for analysis logic; (4) handle large inputs efficiently.

8.1 From scripts to tools

Lucas's counting scripts grow into a small tool that reads sign-in and mail exports, joins them by user and time, and prints suspicious combinations. Functions separate reading, analysis and output.

The separation meant that when the mail service changed its export format halfway through term, only the reading function needed changing; the analysis and output parts stayed the same. Lucas also added a command-line option for the time window, so analysts could rerun the tool for a different period without editing the code itself.

8.2 Dates and times

The datetime module parses timestamps and converts zones. Mixing local and UTC times is the most common bug in SOC scripts.

When sign-in and mail records come from different services, one may record UTC and the other local time, and during British Summer Time the two differ by an hour. A rule that links events within sixty minutes would then match the wrong pairs or none. Lucas converts every timestamp to a timezone-aware datetime in UTC as it is read, so comparisons later in the tool never need to think about zones.

8.3 JSON

Cloud services export JSON. The json module reads nested records; Lucas writes small functions to extract the fields he needs.

The mail service's rule export nested the rule's conditions and actions several levels deep, and some rules lacked fields others had. Lucas's extract function returns a default when a field is missing, rather than raising an error, and his tests include a rule with no forwarding address so that case is checked. Fields he does not need are ignored, which keeps the function short and makes later format changes less likely to break it.

8.4 Unit tests

unittest checks functions with known inputs: "a rule created 59 minutes after a risky sign-in should match; 61 minutes should not".

Boundary cases like these catch the mistakes most likely to slip through: a comparison written as less-than instead of less-than-or-equal would pass ordinary examples and fail only at exactly sixty minutes. Lucas added a test for that boundary too. When a classmate later refactored the matching function, the tests showed within seconds that the behaviour had not changed. Tests also document intent for the next reader.

8.5 Efficiency

Reading files line by line and using dictionaries for lookups keeps memory small and speed high on large exports.

Lucas's first version loaded every sign-in into a list and searched it for each rule, which took minutes on a month's export. Building a dictionary keyed by user once, then looking up each rule's user, cut the run to a few seconds. The change mattered during the incident, when analysts needed to rerun the tool repeatedly as new data arrived during the day.

8.6 Errors

The tool logs records it cannot parse and continues; at the end it prints totals read, skipped and matched.

During the phishing incident, the tool reported that a few hundred of several hundred thousand sign-in records were skipped, all with a malformed timestamp from one legacy application. Because the count was printed, Lucas could check that none of the skipped records belonged to s2047 or s3310 before trusting the result. Silent skipping would have hidden that question entirely, along with the faulty application.

8.7 Sharing

A README explains inputs, outputs and limits, so other student analysts can run it.

The README lists the expected export files, the command to run, an example of the output and known limits, such as treating unknown time zones as UTC. When a student analyst on the night shift used the tool for the first time during the incident, the README was enough; nobody had to wake Lucas to ask how it worked. The README is updated in the same change as the code.

Configuration

Thresholds such as the 60-minute window live in a small configuration file, not inside the code. Changing a threshold then needs no code change, and the configuration is reviewed like code. Lucas records which configuration produced each result so findings can be reproduced later.

Peer review of tools

Another student analyst runs the tool on a test export before it is used in a case.

Worked example

A unit test fails: a rule created at 02:59 after a sign-in at 02:00 is not matched. Lucas finds he compared minutes only, ignoring hours. He switches to datetime objects and the test passes. Without the test, the bug would have hidden matches spanning an hour boundary.

Synthetic case. Code and data are synthetic.

Errors and limits

Tests cover the cases you imagine; add new tests when real data surprises you.

Exercises

EX022 — Time bug. What caused the failed test, and what fixed it?
Show marking criteria
Rubric: 3 = compared minutes ignoring hours; use datetime objects; 2 = partial; 1 = unknown; 0 = none.
EX023 — Totals output. Why print totals read, skipped and matched?
Show marking criteria
Rubric: 3 = makes data loss and coverage visible; 2 = partial; 1 = habit; 0 = none.
EX024 — README contents. What should the tool's README explain?
Show marking criteria
Rubric: 3 = inputs, outputs, limits; 2 = two; 1 = one; 0 = none.

Synthesis

Structured, tested tools built on the standard library make SOC analysis repeatable and shareable.

Continue studying — chapter 08 of 40

← Chapter 07 (draft) · Course chapter index · Chapter 09 (draft) →