Chapter 41 — Security operations and blue team: system optimization
Lesson map
Lesson 41 published. Map, example, EX121.
flowchart TD
N0["Learning outcomes"]
N1["41.1 What to optimise"]
N2["41.2 Finding bottlenecks"]
N3["41.3 Data-driven targets"]
N4["41.4 Local versus global"]
N5["41.5 Automation priorities"]
N6["41.6 Post-deployment review"]
N7["41.7 Diminishing returns"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX121"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["41.1 What to optimise"]
N2["41.2 Finding bottlenecks"]
N3["41.3 Data-driven targets"]
N4["41.4 Local versus global"]
N5["41.5 Automation priorities"]
N6["41.6 Post-deployment review"]
N7["41.7 Diminishing returns"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX121"]
[Learning outcomes]
|
v
[41.1 What to optimise]
|
v
[41.2 Finding bottlenecks]
|
v
[41.3 Data-driven targets]
|
v
[41.4 Local versus global]
|
v
[41.5 Automation priorities]
|
v
[41.6 Post-deployment review]
|
v
[41.7 Diminishing returns]
|
v
[Worked example] --> [Exercise EX121]
Learning outcomes
You will (1) optimise SOC performance across detection quality, speed and cost; (2) use data to choose optimisation targets; (3) avoid local optimisations that harm the whole; (4) evaluate optimisations after deployment.
41.1 What to optimise
The SOC balances detection quality, time to respond, analyst load and cost. Improving one can harm another: cheaper storage slows investigations; more rules raise load.
Optimising one measure in isolation can make the SOC worse overall. A team that only cut cost might move all logs to slow storage and lose hours in each investigation. Naming the four measures together keeps trade-offs visible, and proposals must state their expected effect on each. Amara writes the four measures at the top of every optimisation proposal, with current values, so readers can see what is at stake.
41.2 Finding bottlenecks
Laboratory II showed acknowledgement, not detection, limited response time. Optimising rule speed further would have changed nothing. Optimisation starts at the bottleneck.
In any chain, the slowest stage limits the whole. The rule took about four minutes; acknowledgement took up to eleven. Halving rule time would save two minutes at best, while fixing acknowledgement saved far more. Measuring each stage before optimising avoids spending effort where it cannot help. Amara compares the stage times side by side in one table before suggesting any change, and she circles the slowest stage in each run.
41.3 Data-driven targets
Amara ranks candidate improvements by expected effect on the 90th-percentile response time and analyst minutes per day, using measured data from earlier chapters.
Ranking by expected effect uses numbers already gathered: stage times from the laboratory, alert volumes from the night-load study and handling times from triage records. Each candidate gets an estimate with its assumptions. The ranking guides where to start, and actual results later show how good the estimates were. Paolo reviews the assumptions before work starts on any of them.
41.4 Local versus global
Moving rules to a morning queue optimised night load but could slow response to those alerts. The team tracks the moved rules' outcomes to make sure the global effect is positive.
A change can look good in its own area while causing harm elsewhere. Moving rules to the morning lightened the night, but if one of those rules ever caught a real attack at night, the delay could matter. Tracking outcomes of the moved rules checks that the overall result is still positive.
41.5 Automation priorities
Automated enrichment saved two minutes per alert; automating customer phone lookup saves another minute for takeover alerts. Small savings on frequent tasks add up.
A minute saved on an alert that occurs twice a month barely matters; a minute saved on one that occurs fifty times a day adds up to hours each week. Phone lookup for takeover alerts is frequent and repetitive, so automating it is worth the effort. Amara lists candidates by frequency times time saved. Phone lookup goes first.
41.6 Post-deployment review
Each optimisation is reviewed after a month: did the metric move, and did anything else get worse?
A month gives enough data to see whether the targeted metric moved. Checking other metrics catches side effects, such as faster acknowledgement but more errors in closure reasons. If the change did not help, it is rolled back or adjusted rather than kept because it was planned. The review is a short written note with the before-and-after figures, signed off by Paolo. Notes for changes that were rolled back are kept too, since they record what did not work and stop the same idea being retried without new evidence.
41.7 Diminishing returns
After the main bottlenecks are fixed, further gains cost more; at some point, effort is better spent on validation gaps.
The first fixes remove large, obvious delays; later ones chase seconds at rising cost. Once response time is well within target, effort may do more good closing a technique that currently leaves no trace. Choosing when to stop optimising is itself an engineering decision, made with data. Amara proposes a simple rule: once the 90th percentile sits well inside target for three months, optimisation pauses and validation gaps take priority.
Cost optimisation
Cost is optimised last and separately: licence tiers, storage classes and retention are reviewed yearly. Savings are reinvested only after checking they do not reduce detection or investigation capability, which Amara documents in the review.
Worked example
Amara's optimisation review: night alert minutes down 41% (target 40%); p90 acknowledgement for takeover alerts 12.8 minutes (target 15); no missed priority incidents; morning queue averaging 22 minutes of work. She recommends shifting effort to the depot logging gap from validation.
Synthetic case. Figures are invented.
Errors and limits
One month of data may include unusual weeks; continue tracking.
Exercises
- EX121 — Bottleneck first. Why start optimisation at the bottleneck? Rubric: 3 = improving non-bottleneck stages does not change overall result; 2 = partial; 1 = vague; 0 = none.
- EX122 — Local effect check. How did the team ensure the morning-queue change helped overall? Rubric: 3 = tracked moved rules' outcomes for slower response harm; 2 = partial; 1 = assumed; 0 = none.
- EX123 — Next effort. Why did Amara recommend moving to the depot logging gap? Rubric: 3 = targets met; diminishing returns; validation gap more valuable; 2 = partial; 1 = random; 0 = none.
Synthesis
Optimise the whole system starting at its bottleneck, check for side effects, and know when to switch effort elsewhere.
Continue studying — chapter 41 of 50
← Chapter 40 (draft) · Course chapter index · Chapter 42 (draft) →