Chapter 36 — Performance and scale of event pipelines
Lesson map
Lesson 36 published. Map, example, EX071.
flowchart TD
N0["Learning outcomes and connection"]
N1["36.1 Capacity basics"]
N2["36.2 When the buffer is full"]
N3["36.3 Delay changes decisions"]
N4["36.5 Testing performance"]
N5["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX071"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["36.1 Capacity basics"]
N2["36.2 When the buffer is full"]
N3["36.3 Delay changes decisions"]
N4["36.5 Testing performance"]
N5["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX071"]
[Learning outcomes and connection]
|
v
[36.1 Capacity basics]
|
v
[36.2 When the buffer is full]
|
v
[36.3 Delay changes decisions]
|
v
[36.5 Testing performance]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX071]
Learning outcomes and connection
After this chapter you can (1) model an event pipeline's capacity with arrival rate, service rate and buffering; (2) use Little's law to relate backlog, throughput and delay; (3) compare loss and backpressure policies under an incident spike; and (4) test whether delayed events change case severity. It builds on Chapters 1, 6, 32 and 34 and prepares Chapters 41 and 42.
36.1 Capacity basics
A pipeline stage with service rate $\mu$ and arrival rate $\lambda$ is stable only if $\lambda < \mu$ on average. Short spikes above $\mu$ are absorbed by buffers; the buffer size determines how long a spike can last before something gives. Little's law, $L = \lambda W$, relates the average number in the system $L$, throughput $\lambda$ and average time in system $W$: a backlog of 9,000 events at a throughput of 1,000 per minute means about 9 minutes of delay. Batching improves throughput (fewer per-event overheads) but adds delay up to the batch interval.
36.2 When the buffer is full
Two policies: drop (discard new or old events when full) or backpressure (slow or stop the producer). Dropping loses evidence; backpressure moves the queue upstream — into collectors, which have their own limits (Chapter 3), or into the source, which may then drop silently. Either way, the loss or delay must be visible: drop counters, queue depths and event-time lag per source are first-class metrics. Storage is the third budget: at 1 KB per event, 1,000 events/s is about 86 GB/day before compression and indexes.
36.3 Delay changes decisions
Correlation rules use time windows. If they evaluate on arrival time, delayed events fall outside the window and the rule fails to fire; if they evaluate on event time, the rule must wait for late events (a watermark), adding latency. The trade-off is explicit: wait longer and decide later, or decide sooner and accept missed correlations — then re-evaluate when late events arrive.
Lab 36 puts both sides of this trade-off in numbers. With the middle event 8 minutes late, the arrival-time span grows to 10 minutes and the rule stays silent. Evaluated on event time, the span is still 4 minutes and the rule escalates. HVSS's 10-minute watermark means such escalations can arrive up to 10 minutes later than a perfect pipeline would deliver them.
36.4 Worked example (synthetic): HVSS incident spike
"""Lab 36 (synthetic): pipeline under an incident spike; bounded buffer, drop vs backpressure, late events."""
def sim(policy, cap=3000, mu=1000, steps=60):
buf, lost, late, maxdelay = [], 0, 0, 0
for t in range(steps): # one step = 1 minute
arrivals = 1500 if 10 <= t < 25 else 600
for i in range(arrivals):
if len(buf) < cap or policy == "backpressure": buf.append(t)
else: lost += 1
served = buf[:mu]; buf = buf[mu:]
for a in served:
d = t - a; maxdelay = max(maxdelay, d)
if d > 5: late += 1
return lost, late, maxdelay, len(buf)
for pol in ("drop", "backpressure"):
lost, late, md, rem = sim(pol)
print(f"{pol:12s} lost={lost:5d} late(>5 min)={late:5d} max delay={md:2d} min remaining buffer={rem}")
# Does lateness change severity? A rule escalates if 3 related events fall within an 8-minute window.
events = [0, 2, 4] # true times (minutes) of three related events
for delay in (0, 4, 8):
seen = [events[0], events[1] + delay, events[2]] # middle event delayed
by_arrival = max(seen) - min(seen) <= 8
by_event_time = max(events) - min(events) <= 8
print(f"middle event delayed {delay} min: arrival-time span {max(seen)-min(seen)} -> escalate={by_arrival}; event-time span {max(events)-min(events)} -> escalate={by_event_time}")
Expected output:
drop lost= 5500 late(>5 min)= 0 max delay= 2 min remaining buffer=0
backpressure lost= 0 late(>5 min)= 9000 max delay= 8 min remaining buffer=0
middle event delayed 0 min: arrival-time span 4 -> escalate=True; event-time span 4 -> escalate=True
middle event delayed 4 min: arrival-time span 6 -> escalate=True; event-time span 4 -> escalate=True
middle event delayed 8 min: arrival-time span 10 -> escalate=False; event-time span 4 -> escalate=True
During a 15-minute spike at 1,500 events/min against capacity 1,000/min, the drop policy with a 3,000-event buffer loses 5,500 events but delays nothing by more than 2 minutes. Backpressure loses nothing but delays 9,000 events by more than 5 minutes, up to 8 minutes. The second part shows the consequence for detection: a three-event rule with an 8-minute window evaluated on arrival time stops firing when the middle event is 8 minutes late, while event-time evaluation still fires. HVSS chose backpressure with event-time evaluation and a 10-minute watermark for correlation rules, plus a dedicated fast path that bypasses the shared queue for a small set of critical single-event detections.
36.5 Testing performance
Load tests replay recorded (sanitised) or synthetic traffic at multiples of normal rate — 2×, 5×, 10× — and measure throughput, event-time lag percentiles, loss and rule outcomes. Compare rule outcomes against the same traffic replayed at normal rate: any difference is a performance-induced detection change.
For HVSS, the decisive check is that the three-event rule from Lab 36 still escalates at 5× load as it does at normal rate. Throughput alone would not show whether backpressure delays had pushed the middle event outside an arrival-time window.
36.6 Errors and limits
- Sizing for average rather than incident peak.
- Silent loss at buffers.
- Correlation on arrival time.
- Load tests that measure throughput but not detection outcomes.
Lab 36 shows the consequence of each of these errors in turn. A pipeline sized for the 600 events per minute of normal traffic meets 1,500 during the spike and must either lose or delay. Under the drop policy, 5,500 events disappear unless a drop counter reports them. On arrival time, an 8-minute delay is enough to suppress the escalation. A load test that reported only the 1,000 events per minute served would have missed that changed rule outcome.
Exercises — chapter 36
EX071 — Little's law. Objective: estimate delay. Inputs: a stage processes 1,200 events/s; monitoring shows an average queue of 90,000 events. Steps: estimate average waiting time; estimate how long a 3× spike can last before a 500,000-event buffer is full, starting from 90,000. Expected evidence: two numbers. Rubric: 3 = 75 s and ≈ 171 s (excess 2,400/s); 2 = one; 1 = method; 0 = none.
EX072 — Policy choice. Objective: argue a design. Inputs: Lab 36 results. Steps: recommend a policy for (a) authentication events used for impossible-travel rules and (b) verbose debug logs; justify with loss, delay and decision effects. Expected evidence: recommendation. Rubric: 3 = backpressure/no-loss with event-time evaluation for (a), sampling or drop-oldest with counters for (b), with stated consequences; 2 = partial; 1 = one policy for all; 0 = none.
Chapter synthesis
Pipelines have rates, buffers and budgets. Spikes force a choice between loss and delay; make both visible, evaluate correlations on event time with watermarks, and load-test detection outcomes, not just throughput.
Continue studying — chapter 36 of 50
← Chapter 35 · Course chapter index · Chapter 37 →
Continue to the next authored lesson.