Chapter 39 — Security operations and blue team: maintainability | Engineering Security operations and blue team

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 39 published. Map, example, EX115.

flowchart TD
  N0["Learning outcomes"]
  N1["39.1 Maintenance costs"]
  N2["39.2 Rule lifecycle"]
  N3["39.3 Readable rules"]
  N4["39.4 Shared components"]
  N5["39.5 Toil"]
  N6["39.6 Documentation"]
  N7["39.7 Knowledge transfer"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX115"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["39.1 Maintenance costs"]
  N2["39.2 Rule lifecycle"]
  N3["39.3 Readable rules"]
  N4["39.4 Shared components"]
  N5["39.5 Toil"]
  N6["39.6 Documentation"]
  N7["39.7 Knowledge transfer"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX115"]

[Learning outcomes]
    |
    v
[39.1 Maintenance costs]
    |
    v
[39.2 Rule lifecycle]
    |
    v
[39.3 Readable rules]
    |
    v
[39.4 Shared components]
    |
    v
[39.5 Toil]
    |
    v
[39.6 Documentation]
    |
    v
[39.7 Knowledge transfer]
    |
    v
[Worked example] --> [Exercise EX115]

Learning outcomes

You will (1) design detections and pipelines that remain maintainable; (2) manage rule lifecycle and retirement; (3) document for future engineers; (4) reduce operational toil.

39.1 Maintenance costs

Every rule, parser and integration costs time to maintain. A SOC with 400 rules nobody understands is weaker than one with 150 well-kept rules.

Each rule needs someone to check that its data source still exists, that its fields have not been renamed and that its threshold still suits current traffic. Multiply that by hundreds and a small team cannot keep up, so rules decay quietly. Fewer, better-understood rules mean each gets attention, analysts trust what fires, and gaps are easier to see when the set is reviewed against the threat analysis.

39.2 Rule lifecycle

Rules move through proposed, testing, monitor-only, active, under review and retired. Each state has entry criteria. Rules that never fire or never lead to action are reviewed for retirement.

Entry criteria stop rules from skipping steps: a rule cannot become active without a passing test and a week in monitor-only mode with acceptable results. 'Under review' is a holding state for rules whose data source changed or whose precision fell. A rule that has not fired in a year may be protecting against something rare, so retirement needs a reason, not just silence, and is approved by Paolo.

39.3 Readable rules

Clear names, comments explaining intent, and references to the requirement and ATT&CK technique let others maintain rules when the author leaves.

A rule named 'portal_burst_v3_final' tells a maintainer little; 'portal possible takeover: failures then success from one network' tells them what it is for. Comments explain choices, such as why the window is ten minutes. References link the rule to R-07 and the relevant technique, so a reviewer can judge whether a proposed change still serves the original need.

39.4 Shared components

Common logic, such as "is this address in a depot subnet?", lives in one shared lookup rather than copied into twenty rules, so a change happens in one place.

If twenty rules each contain their own list of depot subnets, adding a new depot means editing twenty files, and one will be missed. A shared lookup fed from the asset inventory updates every rule at once. Shared components need tests and an owner of their own, since a mistake there affects every rule that uses them.

39.5 Toil

Toil is repetitive manual work that scales with load: restarting a stuck collector by hand, copying alert fields into tickets. Engineers automate toil to free analysts.

Toil grows with the business: more depots mean more collectors to restart, more alerts mean more fields to copy. It also wears people down and invites mistakes. Amara measures how many minutes a week analysts spend copying alert fields into tickets and finds it is over three hours. Automating the copy is a small engineering task with a clear return.

39.6 Documentation

Runbooks, architecture records and rule notes are kept beside the code and reviewed with changes.

Documentation kept elsewhere drifts away from what the code actually does. Keeping it beside the code means a reviewer sees both together and can ask why a runbook was not updated when the rule changed. Review checklists include a documentation line, so changes without updated notes are sent back. Runbooks are tested by someone other than their author at least once a year: they follow the steps on a test system and note anything unclear or out of date.

39.7 Knowledge transfer

Before Amara's placement ends, she runs a handover session and pairs Paolo's next junior with her on two rules.

A handover session explains how the depot logging works, where its weak points are and which decisions are still open. Pairing on real rules teaches more than reading notes, because the junior asks questions as problems arise. Amara records the session and updates the runbooks with every question the junior asked that the documentation did not answer. Paolo attends the session and signs off the handover list.

Dependency tracking

Each rule lists the data sources and lookups it depends on. When a system is decommissioned, a query over these lists shows which rules are affected, so retirement happens deliberately rather than leaving rules that silently never fire.

Worked example

Amara's quarterly rule review finds 31 rules that have not fired in a year. Twelve cover techniques still relevant and are verified with test events — they stay. Fifteen depend on a decommissioned system and are retired. Four duplicate newer rules and are merged. The review notes why each decision was made.

Synthetic case. Counts are invented.

Errors and limits

A rule that never fires may still be valuable if the behaviour is rare and serious; test before retiring.

Exercises

EX115 — Lifecycle states. List the rule lifecycle states.
Show marking criteria
Rubric: 3 = proposed, testing, monitor-only, active, under review, retired; 2 = four; 1 = two; 0 = none.
EX116 — Quarterly review. What happened to the 31 silent rules?
Show marking criteria
Rubric: 3 = 12 kept after tests, 15 retired, 4 merged; 2 = two; 1 = one; 0 = none.
EX117 — Toil. Define toil and give an example.
Show marking criteria
Rubric: 3 = repetitive manual work that scales with load; e.g. restarting collectors; 2 = partial; 1 = vague; 0 = none.

Synthesis

Maintainable SOCs keep fewer, clearer rules with defined lifecycles, shared components and little toil.

Continue studying — chapter 39 of 50

← Chapter 38 (draft) · Course chapter index · Chapter 40 (draft) →