Chapter 14 — Software engineering | Engineering Security operations and blue team

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 14 published. Map, example, EX040.

flowchart TD
  N0["Learning outcomes"]
  N1["14.1 Detection as code"]
  N2["14.2 Requirements"]
  N3["14.3 Testing"]
  N4["14.4 Review"]
  N5["14.5 Release and rollback"]
  N6["14.6 Technical debt"]
  N7["14.7 Documentation"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX040"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["14.1 Detection as code"]
  N2["14.2 Requirements"]
  N3["14.3 Testing"]
  N4["14.4 Review"]
  N5["14.5 Release and rollback"]
  N6["14.6 Technical debt"]
  N7["14.7 Documentation"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX040"]

[Learning outcomes]
    |
    v
[14.1 Detection as code]
    |
    v
[14.2 Requirements]
    |
    v
[14.3 Testing]
    |
    v
[14.4 Review]
    |
    v
[14.5 Release and rollback]
    |
    v
[14.6 Technical debt]
    |
    v
[14.7 Documentation]
    |
    v
[Worked example] --> [Exercise EX040]

Learning outcomes

You will (1) apply requirements, design, testing and release practices to detection content; (2) use version control and code review for rules; (3) plan deployment and rollback; (4) manage technical debt in a SOC.

14.1 Detection as code

Detection rules are software. Writing them as text files in a standard format such as Sigma, storing them in version control and reviewing changes brings software engineering discipline to the SOC.

Version control records who changed a rule, when and why, which answers questions after an incident such as whether a detection was active at the time. A standard format like Sigma also makes rules portable between SIEM products, and lets teams share and review rules using ordinary software tools rather than a vendor console.

14.2 Requirements

Each rule starts with a requirement: what behaviour, on which assets, with what acceptable false-positive rate and response. Requirements trace to threats, often expressed as ATT&CK techniques.

Writing the requirement before the rule keeps the work aimed at a real risk. Stating an acceptable false-positive rate up front gives the test a pass mark. Naming the response tells the rule's author how much detail an alert must carry: an alert that triggers an account lock needs stronger evidence than one that only feeds a daily review.

14.3 Testing

Rules are tested against recorded benign data (to measure false positives) and against safe test events (to confirm they fire). Tests run automatically when a change is proposed.

Recorded benign data might be a week of normal portal logs; safe test events are synthetic records crafted to look like the target behaviour without any real attack. A rule that fires on the synthetic burst but stays quiet on the normal week passes. Running both automatically on every change catches regressions before deployment.

14.4 Review

A second engineer reviews logic, performance and documentation. Review catches overly broad conditions before they flood analysts.

Performance is easy to overlook: a rule that searches every record for a substring might work on test data but strain the SIEM in production. The reviewer also checks that documentation tells an analyst what to do when the rule fires. Paolo uses a short checklist so reviews are consistent across reviewers and rules. Review comments and the author's responses stay attached to the change in version control.

14.5 Release and rollback

New rules deploy first in a "monitor only" mode that records matches without alerting. After a week of acceptable results, they move to alerting. If problems appear, the previous version is restored from version control.

Monitor mode shows how often a rule would fire on live data without adding to the analysts' queue. Acceptable results mean the match rate and precision meet the requirement. Because every version lives in version control, rollback is quick and leaves a record, unlike editing a rule in place in a console with no history.

14.6 Technical debt

Old, untested rules accumulate. Paolo schedules a quarterly review to retire rules that no longer match any asset or threat.

A rule written for a server decommissioned two years ago still consumes processing and may confuse new analysts. Rules whose tests no longer pass, or that have no tests at all, carry hidden risk. The quarterly review lists each rule with its last match, test status and owner, and retired rules are kept in version history rather than deleted outright.

14.7 Documentation

Each rule carries a description, author, references, known benign triggers and response guidance.

Known benign triggers save analysts time: if a backup job routinely trips a rule, the documentation says so and how to recognise it. References link the rule to the threat it addresses. Response guidance gives the first steps, so a newer analyst can act correctly at night without waiting for the rule's author. A rule missing any of these fields fails the automated checks and cannot be merged, so documentation is never left for later and forgotten.

Ownership

Every rule has a named owner who answers questions and reviews its performance.

Worked example

Amara writes her portal stuffing rule as a Sigma-style file with a requirement link, two test cases and response guidance. In monitor-only mode it matches four times in a week: two real bursts and two load tests by the web team. She adds an exception tied to the load-test tool's signed user agent and a calendar check, then promotes the rule.

Synthetic case. The rule and week are invented.

Errors and limits

Process overhead can slow urgent detections; keep an emergency path with later review.

Exercises

EX040 — Monitor-only mode. What is monitor-only deployment, and why use it?
Show marking criteria
Rubric: 3 = records matches without alerting to measure noise first; 2 = partial; 1 = no alerting ever; 0 = none.
EX041 — Load-test exception. How were load-test matches handled?
Show marking criteria
Rubric: 3 = exception tied to signed user agent and calendar check; 2 = partial; 1 = deleted rule; 0 = none.
EX042 — Rule documentation. List four things each rule document contains.
Show marking criteria
Rubric: 3 = description, author, references, benign triggers, response guidance (four); 2 = three; 1 = one; 0 = none.

Synthesis

Detection-as-code applies requirements, tests, review and staged release to keep rules reliable.

Continue studying — chapter 14 of 50

← Chapter 13 (draft) · Course chapter index · Chapter 15 (draft) →