Chapter 36 — Security operations and blue team: performance and scale
Lesson map
Lesson 36 published. Map, example, EX106.
flowchart TD
N0["Learning outcomes"]
N1["36.1 Events per second"]
N2["Lab — sizing the pipeline"]
N3["36.2 Reading the result"]
N4["36.3 Peaks"]
N5["36.4 Reducing volume"]
N6["36.5 Query performance"]
N7["36.6 Scale tests"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX106"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["36.1 Events per second"]
N2["Lab — sizing the pipeline"]
N3["36.2 Reading the result"]
N4["36.3 Peaks"]
N5["36.4 Reducing volume"]
N6["36.5 Query performance"]
N7["36.6 Scale tests"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX106"]
[Learning outcomes]
|
v
[36.1 Events per second]
|
v
[Lab — sizing the pipeline]
|
v
[36.2 Reading the result]
|
v
[36.3 Peaks]
|
v
[36.4 Reducing volume]
|
v
[36.5 Query performance]
|
v
[36.6 Scale tests]
|
v
[Worked example] --> [Exercise EX106]
Learning outcomes
You will (1) size SIEM ingestion and storage from source rates; (2) plan for peaks; (3) measure query performance; (4) decide what to filter, summarise or tier.
36.1 Events per second
Ingestion is measured in events per second (EPS). Summing sources gives the average; peaks during incidents or business hours can be several times higher.
Average EPS sets the baseline for storage and licensing; peak EPS sets the size of collectors and processing. A system sized only for the average will lag or drop events exactly when activity spikes, often during an incident. Engineers measure EPS per source over several weeks to see daily, weekly and seasonal patterns. Amara records results per source in a table.
Lab — sizing the pipeline
"""Events-per-second and storage sizing for a SIEM (synthetic sources)."""
sources = {"portal_web": 900, "windows_events": 1400, "firewall": 2600, "dns": 1100} # average EPS
bytes_per_event = 550
peak_factor = 2.5
eps = sum(sources.values())
gb_day = eps * bytes_per_event * 86400 / 1e9
print(f"average EPS={eps} peak EPS={eps*peak_factor:.0f}")
print(f"raw GB/day={gb_day:.1f} 90-day hot TB={gb_day*90/1000:.1f}")
Run with python3 lab_capacity.py. Verified output:
average EPS=6000 peak EPS=15000
raw GB/day=285.1 90-day hot TB=25.7
36.2 Reading the result
Four sources give 6,000 EPS on average and 15,000 at a 2.5× peak. At 550 bytes per event, raw volume is 285.1 GB per day, and 90 days hot would need 25.7 TB — far above the 12 TB tier, confirming the need for the tiering decided in chapter 26.
The arithmetic: 6,000 events per second times 86,400 seconds times 550 bytes gives about 285 GB a day. Ninety days of that is far beyond the hot tier. Peak rate matters for processing but not for storage, since storage follows the daily total. The comparison makes tiering a necessity rather than an option.
36.3 Peaks
Pipelines must handle peaks without dropping priority data. Buffers absorb short spikes; sustained peaks need capacity or graceful degradation.
A buffer acts like a reservoir: it fills during a spike and drains when the rate falls. If a peak lasts longer than the buffer can hold, events must be dropped or processed more slowly. Graceful degradation decides which events go first, keeping authentication and other priority sources intact. Buffer sizes are set from the longest spikes measured on each source over the past year, with a margin, and the margin is checked again after any major new source joins.
36.4 Reducing volume
Options: filter events with no detection or investigation value, summarise high-volume sources (for example, firewall flows into hourly aggregates), and tier storage. Each reduction is checked against detection and investigation needs before applying.
Filtering removes events entirely, so it needs the most care: something that seems useless now may matter in a later investigation. Summarising keeps the pattern but loses detail, such as individual flows. Tiering keeps everything but moves older data to cheaper storage. Each choice is recorded with the reason and the checks done.
36.5 Query performance
Rules that scan large time ranges or unindexed fields slow everything. Engineers measure rule run times and rewrite the slowest.
A rule that searches thirty days of firewall logs every five minutes repeats enormous work and competes with analysts' searches. Engineers list rules by run time and resource use, then rewrite the worst, for example by narrowing the time range or using indexed fields. The improvement is measured before and after. A run-time report is produced weekly and the slowest five rules are named in it.
36.6 Scale tests
Before peak season, the team replays a day of synthetic logs at 3× speed in the test SIEM to check that rules keep up.
Replaying at three times normal speed simulates a heavy peak without waiting for one. The test watches rule delays, queue lengths and dropped events. If rules fall behind, the team adds capacity or optimises queries before the busy season, not during it. Results are compared with the previous year's test. The synthetic day includes a simulated incident burst, so the test reflects the uneven load of a real busy day.
Cost visibility
Amara publishes the monthly cost per source so owners see what their logs cost. Teams that send verbose debug logs often reduce them once they see the bill, without any SOC request.
Worked example
At this volume the earlier tiering, 90 days in the SIEM with firewall and DNS moving to the lake after 14 days, no longer fits, so Amara tightens it: her plan keeps portal, Windows and DNS events hot for 30 days and moves firewall events to the lake after 7 days with hourly summaries kept hot. Hot storage falls to about 10 TB. Before applying, she checks that no detection uses raw firewall events older than 7 days.
Synthetic case. Rates and sizes come from the lab script.
Errors and limits
Bytes per event vary by source; measure each source rather than using one average.
Exercises
- EX106 — Sizing result. Report average EPS, peak EPS, GB/day and 90-day hot TB. Rubric: 3 = 6,000; 15,000; 285.1; 25.7; 2 = three; 1 = one; 0 = none.
- EX107 — Volume reduction. Name three ways to reduce stored volume. Rubric: 3 = filter, summarise, tier; 2 = two; 1 = one; 0 = none.
- EX108 — Safety check. What did Amara check before tiering firewall events? Rubric: 3 = no detection uses raw firewall events older than 7 days; 2 = partial; 1 = vague; 0 = none.
Synthesis
Size from measured rates, plan for peaks, and reduce volume only after checking detection and investigation needs.
Continue studying — chapter 36 of 50
← Chapter 35 (draft) · Course chapter index · Chapter 37 (draft) →