Chapter 32 — Reliability engineering for response tooling | Engineering Incident response and cyber resilience

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 32 published. Map, example, EX063.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["32.1 Availability arithmetic"]
  N2["32.2 Common cause"]
  N3["32.3 Correlated demand"]
  N4["32.5 Testing the degraded mode"]
  N5["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX063"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["32.1 Availability arithmetic"]
  N2["32.2 Common cause"]
  N3["32.3 Correlated demand"]
  N4["32.5 Testing the degraded mode"]
  N5["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX063"]

[Learning outcomes and connection]
    |
    v
[32.1 Availability arithmetic]
    |
    v
[32.2 Common cause]
    |
    v
[32.3 Correlated demand]
    |
    v
[32.5 Testing the degraded mode]
    |
    v
[Chapter synthesis]
    |
    v
[Worked example] --> [Exercise EX063]

Learning outcomes and connection

After this chapter you can (1) compute availability of response tool chains in series and in parallel; (2) account for common-cause failure in redundancy; (3) design and size a degraded operating mode with queues; and (4) set availability objectives that reflect when the tool is needed most. It builds on Chapters 1, 6, 21 and 23 and prepares Chapters 33, 36 and 40.

32.1 Availability arithmetic

For a repairable component with mean time between failures (MTBF) and mean time to repair (MTTR), steady-state availability is $A = \text{MTBF}/(\text{MTBF} + \text{MTTR})$. Components needed together (in series) multiply: $A_s = \prod A_i$. Redundant components (parallel, any one sufficient) give $A_p = 1 - \prod (1 - A_i)$ if failures are independent. Avižienis et al. (2004) define availability and reliability as dependability attributes. With Lab 32's numbers, the broker's MTBF of 720 h and MTTR of 4 h give 720/724 ≈ 0.99448.

32.2 Common cause

Redundant copies often share software, configuration, credentials or cloud region. The beta-factor model assigns a fraction $\beta$ of each unit's unavailability to causes that defeat all copies. Even a small $\beta$ dominates the result, so diversity and independence matter more than the number of copies.

Lab 32 shows how strongly common cause dominates even with only two brokers. Two independent HVSS brokers would be down about 0.27 h a year, but a β of 0.1 raises that to about 5.08 h. Almost all of the remaining downtime then comes from the shared tenth of failures, such as the bad configuration update pushed to both brokers last year.

32.3 Correlated demand

Response tools are needed most when the environment is failing: an alert broker hosted in the same cloud region as the compromised workload, a paging service reached via the same identity provider under attack. Availability measured over a year can be high while availability during incidents is poor. Objectives should be stated for incident conditions, and verified by exercises (Chapter 40).

At HVSS, the broker outage that Lab 32 sizes for is the case in point. An outage of four hours at 300 alerts per hour is most likely during an incident, when alert volume is high and the shared infrastructure may itself be affected. An exercise that stops the broker under realistic alert load measures that conditional availability; an annual uptime report does not.

32.4 Worked example (synthetic): HVSS alert broker

"""Lab 32 (synthetic): availability of response tooling; redundancy and a degraded mode."""
def avail(mtbf, mttr): return mtbf / (mtbf + mttr)
broker = avail(720, 4)        # alert broker: fails monthly, 4 h repair
pager  = avail(2000, 2)
print(f"broker A = {broker:.5f}  pager A = {pager:.5f}")
series = broker * pager
print(f"series (both needed) A = {series:.5f}  expected downtime/yr = {(1-series)*8760:.1f} h")
par = 1 - (1 - broker) ** 2
print(f"two independent brokers A = {par:.6f}  downtime/yr = {(1-par)*8760:.2f} h")
beta = 0.1   # beta-factor: share of broker failures that hit both copies
par_cc = 1 - ((1 - broker) ** 2 * (1 - beta) + (1 - broker) * beta)
print(f"with beta=0.1 common cause A = {par_cc:.6f}  downtime/yr = {(1-par_cc)*8760:.2f} h")
# degraded mode: during a broker outage at 300 alerts/h, a local spool holds alerts for later replay
rate, outage_h, spool_cap = 300, 4, 1000
lost = max(0, rate * outage_h - spool_cap)
print(f"4 h outage at {rate}/h: spooled {min(rate*outage_h, spool_cap)}, lost {lost}")
print(f"replay at 600/h with 300/h live arrivals drains backlog in {min(rate*outage_h, spool_cap)/(600-rate):.2f} h")

Expected output:

broker A = 0.99448  pager A = 0.99900
series (both needed) A = 0.99348  expected downtime/yr = 57.1 h
two independent brokers A = 0.999969  downtime/yr = 0.27 h
with beta=0.1 common cause A = 0.999420  downtime/yr = 5.08 h
4 h outage at 300/h: spooled 1000, lost 200
replay at 600/h with 300/h live arrivals drains backlog in 3.33 h

A single broker and pager in series lose about 57 hours a year. Two independent brokers would reduce broker-related downtime to a fraction of an hour — but with $\beta = 0.1$ common cause, the pair still loses about 5 hours a year: the shared configuration management pushed a bad update to both last year, so $\beta$ is not hypothetical. The degraded mode is a local spool on each collector: during a 4-hour broker outage at 300 alerts per hour, a 1,000-alert spool keeps the first 1,000 and loses 200 — unacceptable for HVSS's requirement of no alert loss during outages up to 4 h. Sizing the spool to at least 1,200 alerts (plus margin, say 2,000) and replaying at 600/h drains the backlog in about 4 hours after recovery while live alerts continue. The design also needs ordering by severity during replay, so a critical alert spooled at minute 230 does not wait behind 1,000 low-severity ones.

32.5 Testing the degraded mode

A degraded mode that is never exercised does not exist. Test: broker stopped in a staging environment, spool fills, replay occurs, counts reconcile (sent = received + reported lost), latency of replayed alerts measured, analysts see a "degraded" banner. Run quarterly and after changes.

For the HVSS collector spool, the reconciliation target of such a test is concrete. With the spool resized to at least 1,200 alerts, a four-hour staged outage at 300 alerts per hour must show 1,200 sent and 1,200 received, with none reported lost. The replayed backlog should also drain in about four hours at 600 per hour.

32.6 Errors and limits

Assuming independence of redundant copies.
Annual availability as the only measure.
Degraded modes that silently drop data when full.
Replay without severity ordering.

Lab 32 shows the cost of each of these errors in numbers. Assuming independence promises 0.27 h of yearly broker downtime when common cause makes it about 5.08 h. A yearly figure says nothing about the hours when an incident is in progress. The original 1,000-alert spool silently loses 200 alerts in a four-hour outage. Replaying in arrival order would also leave a critical alert spooled at minute 230 queued behind the earlier low-severity ones.

Exercises — chapter 32

EX063 — Size the spool. Objective: design to requirement. Inputs: Lab 32; requirement: no loss for outages up to 6 h at peak 450 alerts/h; replay must finish within 6 h after recovery while live arrivals continue at 300/h. Steps: compute minimum spool size and required replay rate. Expected evidence: numbers and reasoning. Rubric: 3 = 2,700 alerts minimum (with stated margin) and replay ≥ 750/h; 2 = one; 1 = method; 0 = none.

EX064 — Common cause. Objective: reason about diversity. Inputs: $\beta$ = 0.1 result. Steps: list three sources of common cause for HVSS brokers and one mitigation each; recompute availability for $\beta = 0.02$. Expected evidence: list and number. Rubric: 3 = specific causes and availability ≈ 0.99986 (≈ 1.2 h/yr); 2 = partial; 1 = list only; 0 = none.

Chapter synthesis

Compute availability honestly — series multiplies, redundancy helps only to the extent failures are independent, and demand for response tools correlates with failure. Design, size and exercise degraded modes so that outages delay alerts rather than lose them.