◇ OpenCyber Academy
Engineering · Chapter 33 · English

Chapter 33 — Security operations and blue team: safety and resilience

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 33 — Security operations and blue team: safety and resilience

Lesson map

Lesson 33 published. Map, example, EX097.

flowchart TD
  N0["Learning outcomes"]
  N1["33.1 Safety of response"]
  N2["33.2 Critical services"]
  N3["33.3 SOC resilience"]
  N4["33.4 Fallbacks"]
  N5["33.5 Exercises"]
  N6["33.6 Recovery"]
  N7["33.7 Case"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX097"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["33.1 Safety of response"]
  N2["33.2 Critical services"]
  N3["33.3 SOC resilience"]
  N4["33.4 Fallbacks"]
  N5["33.5 Exercises"]
  N6["33.6 Recovery"]
  N7["33.7 Case"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX097"]

[Learning outcomes]
    |
    v
[33.1 Safety of response]
    |
    v
[33.2 Critical services]
    |
    v
[33.3 SOC resilience]
    |
    v
[33.4 Fallbacks]
    |
    v
[33.5 Exercises]
    |
    v
[33.6 Recovery]
    |
    v
[33.7 Case]
    |
    v
[Worked example] --> [Exercise EX097]

Learning outcomes

You will (1) assess safety risks of SOC actions on operations; (2) design response that preserves critical services; (3) plan SOC resilience when its own systems fail; (4) exercise resilience scenarios.

33.1 Safety of response

Response actions can harm the business. Isolating a depot workstation might stop dispatch for a shift. Playbooks state which assets are safety- or business-critical and require approval before isolating them.

A dispatch workstation that stops for a shift could leave dozens of trucks without routes and customers without deliveries. That harm may exceed the risk from a suspicious process, especially if the evidence is weak. Requiring approval for critical assets makes the trade-off explicit: the analyst presents the evidence, the business owner weighs operational impact, and the decision and its reasons are recorded in the case.

33.2 Critical services

Lindqvist lists services that must continue: dispatch, driver routing, customer tracking and payments. For each, the SOC agrees acceptable actions and contacts in advance.

Agreeing actions in advance avoids arguments during an incident, when time is short and stress is high. For dispatch, the agreed action might be blocking one suspicious connection rather than isolating the whole machine. Each service has a named business contact with a phone number that works at night, and the list is checked every quarter because people change roles.

33.3 SOC resilience

The SOC itself can fail: the SIEM may go down, the case system may be unavailable, or the SOC office may be inaccessible. NIST CSF 2.0's Recover function and SP 800-61r3 call for planning and practising recovery.

A SOC that depends entirely on its SIEM is blind when the SIEM is down, which is exactly when an attacker who caused the outage would act. Planning for recovery means listing the SOC's own dependencies, deciding how long each can be missing, and preparing alternatives. Practising those plans matters, because untested recovery steps often fail on the first real attempt.

33.4 Fallbacks

If the SIEM fails, analysts can query critical sources directly using read-only accounts and runbooks. If the case system fails, a shared offline template records actions until it returns.

Read-only accounts let analysts look at authentication and firewall logs on their source systems without risk of changing them. Runbooks give the exact steps, since analysts used to the SIEM's interface may not know each source's own tools. The offline template captures times, actions and decisions, so the case history can be entered into the case system afterwards without gaps.

33.5 Exercises

Twice a year the SOC rehearses: "SIEM unavailable for four hours during a portal attack". Rehearsals reveal missing access, outdated runbooks and unclear roles.

Combining an outage with an attack makes the exercise realistic: real failures often coincide with incidents, sometimes because the attacker caused them. Four hours is long enough for workarounds to be needed rather than waited out. Observers note where analysts hesitated, which steps were missing and which contacts did not answer, and each finding gets an owner and a deadline.

33.6 Recovery

After a failure, the SOC checks for gaps in data, replays buffered logs and reviews whether detections missed anything during the outage.

Logs buffered at sources or collectors during the outage may arrive in a burst when the SIEM returns. Rules that run on live data may not process them, so engineers rerun key detections over the outage window. Any gap that could not be recovered is recorded, so later investigations know that period has incomplete data. Paolo signs off the gap list.

33.7 Case

In the first exercise, analysts discover the read-only accounts for direct log access had expired.

Nobody had noticed because the accounts were never used in normal operations; they expired silently under the password policy. Had the outage been real, analysts would have lost an hour or more getting access approved. The fix is a quarterly check that logs into each fallback account, and an alert two weeks before any of them expires. Amara also adds the fallback accounts to the access review list, so their owners confirm each quarter that they are still needed.

Communication during failure

When SOC systems fail, the SOC tells IT operations and the CISO what visibility is lost and for how long, so others can take extra care.

Worked example

The exercise report lists three findings: expired direct-access accounts (renewed and added to quarterly checks), runbook steps referring to retired servers (updated), and no agreed decision-maker for isolating dispatch systems at night (the operations duty manager is named). The next exercise is scheduled.

Synthetic case. The exercise is invented.

Errors and limits

Exercises are planned; real failures surprise. Keep runbooks simple enough to use under stress.

Exercises

Synthesis

Safe response protects critical services, and a resilient SOC plans and rehearses how to work when its own systems fail.

Continue studying — chapter 33 of 50

← Chapter 32 (draft) · Course chapter index · Chapter 34 (draft) →