Chapter 47 — Failure and recovery trial
Lesson map
Lesson 47 published. Map, example, EX139.
flowchart TD
N0["Learning outcomes"]
N1["47.1 Purpose"]
N2["47.2 Plan"]
N3["Lab — detecting silent sources"]
N4["47.3 Reading the result"]
N5["47.4 Detection in the trial"]
N6["47.5 Recovery"]
N7["47.6 Improvements"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX139"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["47.1 Purpose"]
N2["47.2 Plan"]
N3["Lab — detecting silent sources"]
N4["47.3 Reading the result"]
N5["47.4 Detection in the trial"]
N6["47.5 Recovery"]
N7["47.6 Improvements"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX139"]
[Learning outcomes]
|
v
[47.1 Purpose]
|
v
[47.2 Plan]
|
v
[Lab — detecting silent sources]
|
v
[47.3 Reading the result]
|
v
[47.4 Detection in the trial]
|
v
[47.5 Recovery]
|
v
[47.6 Improvements]
|
v
[Worked example] --> [Exercise EX139]
Learning outcomes
You will (1) plan a controlled failure trial of the monitoring pipeline; (2) detect silent source failures; (3) measure recovery and data loss; (4) improve resilience from trial results.
47.1 Purpose
The trial checks whether the SOC notices when its own data stops, and how well it recovers. Silent failures were ranked high in chapter 23.
Monitoring that fails silently is dangerous because the SOC continues to believe it can see activity it no longer receives. Testing this deliberately is the only way to know whether the safeguards work. The trial also tests recovery: once data flows again, how much of the gap can be filled, and how quickly? Results go into the reliability records.
47.2 Plan
With change approval, the team stops log forwarding from one portal web server for an hour during a quiet period, without telling analysts the exact time. A rollback is ready.
Change approval keeps the trial within Lindqvist's normal process and makes sure IT knows it is deliberate. A quiet period limits the risk of missing a real attack during the gap. Not telling analysts the exact time tests whether the silence check works, rather than whether people remember a scheduled event.
Lab — detecting silent sources
"""Detect log sources that have gone silent compared with their usual cadence (synthetic)."""
now = 600 # minutes since midnight (10:00)
last_seen = {"portal-web-1": 598, "portal-web-2": 599, "portal-web-3": 541, "dc-01": 597, "fw-edge": 520}
expected_gap = {"portal-web-1": 5, "portal-web-2": 5, "portal-web-3": 5, "dc-01": 5, "fw-edge": 120}
for src in sorted(last_seen):
gap = now - last_seen[src]
status = "SILENT" if gap > 3 * expected_gap[src] else "ok"
print(f"{src:13} gap={gap:3} min expected<={expected_gap[src]:3} {status}")
Run with python3 lab_silence.py. Verified output:
dc-01 gap= 3 min expected<= 5 ok
fw-edge gap= 80 min expected<=120 ok
portal-web-1 gap= 2 min expected<= 5 ok
portal-web-2 gap= 1 min expected<= 5 ok
portal-web-3 gap= 59 min expected<= 5 SILENT
47.3 Reading the result
At 10:00, portal-web-3 has been silent for 59 minutes against an expected gap of at most 5, so it is flagged SILENT. The edge firewall's 80-minute gap is normal for its 120-minute cadence. Expected cadence per source avoids false alarms from naturally quiet sources.
Each source is judged against its own normal rhythm. A web server sending events every few seconds is clearly broken after 59 minutes of silence; a firewall that reports every two hours is not broken after 80 minutes. Without per-source cadence, the check would either miss quiet failures or alarm constantly on naturally slow sources.
47.4 Detection in the trial
The silence check runs every 10 minutes and pages on-call. In the trial, it paged 17 minutes after forwarding stopped, because the threshold was three times the expected gap plus the check interval.
The threshold of three times a five-minute gap gives fifteen minutes; with the check running every ten minutes, detection could take up to twenty-five. Seventeen minutes falls in that range, but it means a portal server could be silent for a quarter of an hour before anyone knew. The trial measures this delay precisely so it can be shortened.
47.5 Recovery
After forwarding restarts, buffered events arrive; reconciliation shows 99.2% of the hour's events recovered, with the rest lost when the local buffer filled.
Local buffers hold events while forwarding is stopped and send them when it resumes. The 0.8% lost when it filled shows the buffer was too small for an hour's gap. Reconciliation counts make the loss measurable instead of a guess. Amara notes which events were lost by time range, so investigators know exactly which minutes are incomplete. Her follow-up ticket asks for a larger buffer, sized from the event volume measured during that hour.
47.6 Improvements
Lower the silence threshold for portal servers to two times the gap; enlarge the local buffer to cover two hours.
A lower multiplier shortens detection for portal servers, whose steady flow makes false alarms unlikely. A two-hour buffer covers longer outages without loss. Both changes are tested again in the next trial to confirm they work as intended and do not create new noise. Both changes go through the normal change process with tests, and the threshold change is announced to analysts, who may notice more frequent pages from portal servers at first.
Who knows about the trial
Only the trial lead, on-call engineer and SOC manager know the time; everyone else learns afterwards.
Worked example
Amara's trial report: "Failure noticed after 17 minutes (target 15) — threshold lowered. 0.8% of events lost — buffer enlarged. Re-trial next month." The next trial pages after 12 minutes with no loss.
Synthetic case. Times and figures come from the lab script and invented trials.
Errors and limits
A planned trial at a quiet time is gentler than a real failure during an incident.
Exercises
- EX139 — Silence logic. Why use an expected gap per source? Rubric: 3 = sources have different cadences; avoids false alarms; 2 = partial; 1 = one threshold; 0 = none.
- EX140 — Trial timing. Why did paging take 17 minutes, and what changed? Rubric: 3 = 3× gap threshold plus check interval; lowered to 2×; 2 = partial; 1 = unknown; 0 = none.
- EX141 — Data loss. What data was lost and how was it prevented next time? Rubric: 3 = 0.8% when buffer filled; buffer enlarged to two hours; 2 = partial; 1 = vague; 0 = none.
Synthesis
Deliberately fail parts of the pipeline under control to prove silent failures are noticed and recovery works.
Continue studying — chapter 47 of 50
← Chapter 46 (draft) · Course chapter index · Chapter 48 (draft) →