Chapter 32 — Security operations and blue team: reliability
Lesson map
Lesson 32 published. Map, example, EX094.
flowchart TD
N0["Learning outcomes"]
N1["32.1 Reliability targets"]
N2["32.2 Single points of failure"]
N3["32.3 Redundancy"]
N4["32.4 Graceful degradation"]
N5["32.5 Measuring completeness"]
N6["32.6 Error budgets"]
N7["32.7 Case"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX094"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["32.1 Reliability targets"]
N2["32.2 Single points of failure"]
N3["32.3 Redundancy"]
N4["32.4 Graceful degradation"]
N5["32.5 Measuring completeness"]
N6["32.6 Error budgets"]
N7["32.7 Case"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX094"]
[Learning outcomes]
|
v
[32.1 Reliability targets]
|
v
[32.2 Single points of failure]
|
v
[32.3 Redundancy]
|
v
[32.4 Graceful degradation]
|
v
[32.5 Measuring completeness]
|
v
[32.6 Error budgets]
|
v
[32.7 Case]
|
v
[Worked example] --> [Exercise EX094]
Learning outcomes
You will (1) define reliability targets for the monitoring pipeline; (2) identify single points of failure; (3) design redundancy and graceful degradation; (4) measure availability and data completeness.
32.1 Reliability targets
The pipeline has a target of 99.5% availability and 99.9% completeness of priority sources. Completeness — the share of expected events that arrive — matters as much as uptime, because a running pipeline with missing data still blinds the SOC.
Uptime alone is misleading: a pipeline can be running while silently dropping a share of events. Completeness measures what actually arrives. For priority sources such as authentication logs, 99.9% means at most one event in a thousand lost, which keeps the chance of missing an attack's key event low while remaining achievable with reasonable engineering.
32.2 Single points of failure
Lindqvist's pipeline has one collector per site and one load-balancer log feed. If either fails, a whole source disappears.
One collector per site means a hardware fault there silences every source at that site until it is repaired. The single load-balancer feed means takeover detection depends on one path. Listing these points is the first step; deciding which to remove depends on cost and on the detections each one supports. Amara lists both on the failure analysis with the detections each one supports, which shows the feed carries the most important detection.
32.3 Redundancy
Two collectors per site in active-passive mode, and a second feed path for load-balancer logs, remove the main single points. Redundancy costs money, so it goes first where detection value is highest.
Active-passive means one collector works while the other waits, taking over if the first fails. That is simpler than splitting load between both and avoids duplicate events. Starting with sites and feeds that support the most important detections gives the most protection for each unit of budget, and lower-value sources follow if budget allows.
32.4 Graceful degradation
When the SIEM is overloaded, the pipeline should drop low-value debug logs first, not authentication logs. Priority queues implement this.
When capacity runs short, something must be lost; the question is what. Debug logs are rarely needed for detection, while authentication logs are central. Priority queues let the pipeline drop low-priority events first under pressure. The drop is logged, so the SOC knows which data are missing for that period. Priorities are set per source in the pipeline configuration and reviewed when new sources join.
32.5 Measuring completeness
Comparing events counted at sources with events stored in the SIEM shows loss. A daily reconciliation report lists sources below 99.9%.
Sources can report how many events they sent, and the SIEM knows how many it stored; the difference is loss in between. Where sources cannot report counts, collectors can count what they received instead. The daily report lists each source's ratio, and any below target gets investigated by the source owner. The report goes to source owners each morning and is summarised weekly for Paolo.
32.6 Error budgets
If availability falls below target, new features pause and reliability work takes priority until the budget recovers.
A 99.5% target leaves 0.5% of time as an allowance for downtime. This gives a clear, pre-agreed rule for balancing new work against stability, instead of a fresh argument each time something breaks. The budget resets each quarter; at 99.5%, the allowance is about eleven hours per quarter. Product managers and SOC leads agree this rule in advance, so pausing feature work when the budget is spent is routine rather than a dispute.
32.7 Case
In week one of reconciliation, the portal web servers show 99.95% completeness; the depot firewall shows 97.8% because of a UDP syslog path that drops packets under load.
UDP syslog sends messages without confirmation, so packets dropped under load are simply lost. A 2.2% loss means about one denial in forty-five never reaches the SIEM. Switching the depot firewall to TCP or a forwarding agent with acknowledgement and buffering would fix it, and the reconciliation report will show whether completeness rises above target.
Monitoring the monitors
Collectors, parsers and the SIEM itself report health metrics — queue depth, parse errors, ingestion lag — to a separate monitoring system. If the SIEM is unhealthy, it cannot be trusted to alert on its own failure, so the health alerts go through an independent path to the on-call engineer.
Worked example
Amara moves the depot firewall to TCP-based log forwarding with a local buffer. Completeness rises to 99.96%. She adds the change to the reliability log and keeps the reconciliation report running daily.
Synthetic case. Figures are invented.
Errors and limits
Completeness checks depend on accurate source counts, which some devices do not provide.
Exercises
- EX094 — Completeness. Why is completeness as important as availability? Rubric: 3 = pipeline can run while missing data, still blinding SOC; 2 = partial; 1 = less important; 0 = none.
- EX095 — Degradation order. Under overload, which logs should drop first? Rubric: 3 = low-value debug before authentication; priority queues; 2 = partial; 1 = random; 0 = none.
- EX096 — Firewall fix. What caused the 97.8% completeness, and what fixed it? Rubric: 3 = UDP syslog drops; TCP forwarding with buffer; 99.96%; 2 = partial; 1 = ignored; 0 = none.
Synthesis
Reliable monitoring needs targets for both uptime and completeness, redundancy where value is highest and graceful degradation.
Continue studying — chapter 32 of 50
← Chapter 31 (draft) · Course chapter index · Chapter 33 (draft) →