Chapter 5 — Alternatives for coordinated containment | Postgraduate Incident response and cyber resilience

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 05 published. Map, example, EX021.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["5.1 Coordinated versus piecemeal containment"]
  N2["5.2 Scoping before acting"]
  N3["5.3 Identity first"]
  N4["5.4 Dependencies and blast radius"]
  N5["5.5 Approval and rollback gates"]
  N6["5.6 Execution and verification"]
  N7["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX021"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["5.1 Coordinated versus piecemeal containment"]
  N2["5.2 Scoping before acting"]
  N3["5.3 Identity first"]
  N4["5.4 Dependencies and blast radius"]
  N5["5.5 Approval and rollback gates"]
  N6["5.6 Execution and verification"]
  N7["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX021"]

[Learning outcomes and connection]
    |
    v
[5.1 Coordinated versus piecemeal containment]
    |
    v
[5.2 Scoping before acting]
    |
    v
[5.3 Identity first]
    |
    v
[5.4 Dependencies and blast radius]
    |
    v
[5.5 Approval and rollback gates]
    |
    v
[5.6 Execution and verification]
    |
    v
[Chapter synthesis]
    |
    v
[Worked example] --> [Exercise EX021]

Learning outcomes and connection

You will be able to (1) design containment across cloud and on-premises environments with shared identity; (2) sequence actions to avoid alerting the adversary prematurely or leaving gaps; (3) assess dependencies and blast radius of each action; (4) define approval and rollback gates; and (5) compare two coordinated strategies and recommend one. Chapters 1 and 2 provided decision framing and command; this chapter applies them to the most consequential operational decision in many incidents.

5.1 Coordinated versus piecemeal containment

Piecemeal containment — each team acting as it discovers something — warns the adversary and leaves alternative footholds. Coordinated containment gathers enough scoping information, then executes a planned set of actions within a short window so that the adversary loses access everywhere at once. The trade-off is time: waiting to coordinate allows harm to continue. The commander decides where on this spectrum to sit, using the thresholds of Chapter 1.

Calder's identity compromise shows the trade-off: the big-bang strategy needed 36 hours of preparation, while executive mailboxes were being read throughout.

5.2 Scoping before acting

Coordinated containment needs a scope: compromised identities (users, service principals, keys), hosts, persistence mechanisms, external infrastructure and data at risk. Scope is never complete; decide what level of confidence is "enough" and what monitoring will catch what you missed.

At Calder the scope listed a compromised on-premises domain admin, a synchronized cloud global admin, an OAuth application with mail-read consent, a web shell on the file-transfer server, a partner service principal the adversary may have read, and 300 users whose credentials may have been harvested. Monitoring for old credentials and new persistence was the answer to whatever the list missed.

5.3 Identity first

When identity spans cloud and on-premises (for example, directory synchronization or federation), the adversary can re-enter through either side. Typical sequence: revoke sessions and tokens across the IdP and federated applications; reset credentials (including service-account secrets and, where a directory's key-signing material may be compromised, rotate it according to vendor guidance, which may require two resets separated by replication time); remove rogue federation trusts, application consents and MFA registrations; then block known infrastructure. Doing host cleanup before identity containment invites immediate re-compromise.

5.4 Dependencies and blast radius

Each action has dependants. Rotating a service-account secret used by twenty integrations breaks them unless updated in step; disabling a federation trust may cut off a partner. Build an action-dependency table with owners who will update dependants in the window.

Calder's partner service principal is the example: its secret was rotated with the partner on the call, and a temporary secret was kept for one hour as rollback. Forcing sign-out and password reset for 300 users had a dependant too: the helpdesk had to be staffed for the calls that followed, which is why it appears on the gate checklist.

5.5 Approval and rollback gates

Gates are points where the commander confirms preconditions: scope accepted; dependants ready; communications drafted; monitoring in place. Each action has a rollback plan and a rollback trigger (for example, failure of a critical integration for more than 15 minutes).

Calder's gate checklist named the conditions: scope review signed off, partner engaged, user communication ready, helpdesk staffed, monitoring for old credentials and new persistence deployed, and partner rollback prepared. Its rollback trigger was concrete: if the partner integration was still failing after 30 minutes, the temporary secret would be reinstated under monitoring.

5.6 Execution and verification

Execute in a defined window with a single coordinator calling steps and recording completion times. Verify immediately: no successful authentication with old credentials, no activity from revoked sessions, alerts for re-entry attempts.

At Calder the window was two hours on a weekend, with seven steps called in order, from egress blocks to the 300 user resets. Verification afterwards was what later allowed the Day 3 briefing to say that no attacker activity had been seen for 40 hours, and to back that with an assurance table rather than a hope.

5.7 Worked example (synthetic): identity compromise across cloud and on-premises

Fictional. At Calder Group, an adversary has: a compromised on-premises domain admin account; a cloud global administrator account (synchronized from on-premises); an OAuth application with mail-read consent; and a web shell on an on-premises file-transfer server. A partner integration uses a service principal that the adversary may have read.

Strategy S1 — "big bang" in a 2-hour weekend window: (1) block known adversary IPs and domains at egress; (2) remove web shell and isolate the file-transfer server; (3) disable the OAuth application and revoke its consent; (4) reset the domain admin and cloud global admin, revoke all admin sessions; (5) rotate the directory's key-signing secrets per vendor guidance (twice, with replication interval); (6) rotate the partner service principal secret with the partner on the call; (7) force sign-out and password reset for 300 users whose credentials may have been harvested. Pros: adversary loses all known access simultaneously. Cons: requires 36 hours of preparation; weekend window delays action; large user impact.

Strategy S2 — staged: immediately disable the OAuth app and revoke admin sessions (stops mail reading, the observed harm), then over 48 hours rotate other credentials and remove the web shell. Pros: stops observed harm now. Cons: warns the adversary, who may use the web shell or harvested credentials to establish new persistence.

Gate checklist for S1: scope review signed off; partner engaged; user communication ready; helpdesk staffed; monitoring rules for old credentials and new persistence deployed; rollback for partner integration (temporary secret retained for 1 hour). Rollback trigger: partner integration failing after 30 minutes → reinstate temporary secret under monitoring.

Recommendation (fictional): S1, with one immediate action pulled forward — disabling the OAuth application — because mail reading of executive mailboxes is ongoing harm and the action reveals little about the rest of the plan. This is a hybrid justified by the specific harm and its low signalling cost.

5.8 Common errors and limits

Cleaning hosts before containing identity.
Rotating shared secrets without dependants ready.
No verification after execution.
Treating the plan as final: new scope found during execution needs a decision, not improvisation.
Vendor-specific steps (for example directory key rotation) must follow current vendor guidance.

Calder's plan avoided the first two errors by design: identity steps (admin resets, session revocation, key-signing secret rotation) were in the same window as host clean-up, and the partner secret was rotated with the partner on the call. Pulling the OAuth app disablement forward was a recorded decision, not an improvisation, justified by ongoing reading of executive mailboxes and the action's low signalling cost.

Exercises — chapter 5

Attempt each exercise before opening the worked results. Every case, organization, person, address and number in an exercise is synthetic unless a cited public source is named.

EX021 — Action-dependency table. Steps: build the table for S1 with action, dependants, owner, readiness check and rollback. Rubric: 3 = all seven actions with realistic dependants and rollbacks; 2 = five; 1 = actions only; 0 = none.

EX022 — Sequence critique. Steps: explain why step 2 (web shell removal) before step 4 (admin resets) might be risky, and propose a better order. Rubric: 3 = removal may alert adversary who still holds admin credentials; place host actions in the same window after or simultaneous with identity actions; 2 = partial; 1 = no issue seen; 0 = none.

EX023 — Choose with criteria. Steps: compare S1, S2 and the hybrid on harm stopped, signalling, user impact, preparation time and residual risk; recommend. Rubric: 3 = table and recommendation consistent with criteria and Chapter 1 thresholds; 2 = recommendation without table; 1 = preference only; 0 = none.

EX024 — Verification plan. Steps: list six verification checks for the hour after execution. Rubric: 3 = old-credential authentication attempts, revoked-session activity, new OAuth consents, new federation trusts, web-shell path requests, partner integration health; 2 = four; 1 = two; 0 = none.

EX025 — Gate briefing. Steps: write the 5-minute gate briefing the commander gives before execution. Rubric: 3 = objectives, scope confidence, steps and owners, rollback triggers, communications, go/no-go poll; 2 = missing rollback; 1 = narrative; 0 = none.

Chapter synthesis

Coordinate containment: scope enough, contain identity first, map dependants, gate on readiness, define rollback triggers, execute in one window with a single coordinator, verify immediately — and pull forward only actions that stop ongoing harm at low signalling cost.