Chapter 31 — Implement incident workflow orchestration
Lesson map
Lesson 31 published. Map, example, EX061.
flowchart TD
N0["Learning outcomes and connection"]
N1["31.1 Orchestration vs automation"]
N2["31.2 Timers, retries and escalation"]
N3["31.3 Audit trail"]
N4["31.5 Testing"]
N5["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX061"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["31.1 Orchestration vs automation"]
N2["31.2 Timers, retries and escalation"]
N3["31.3 Audit trail"]
N4["31.5 Testing"]
N5["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX061"]
[Learning outcomes and connection]
|
v
[31.1 Orchestration vs automation]
|
v
[31.2 Timers, retries and escalation]
|
v
[31.3 Audit trail]
|
v
[31.5 Testing]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX061]
Learning outcomes and connection
After this chapter you can (1) implement case workflows as explicit state machines with human tasks; (2) design timeouts, retries and escalations that never convert silence into authority; (3) produce an audit trail sufficient to reconstruct every decision; and (4) test workflow behaviour on timeout, refusal and failure. It builds on Chapters 2, 25 and 27 and prepares Chapters 33, 37 and 39.
31.1 Orchestration vs automation
Security orchestration platforms coordinate tasks across tools and people. The engineering principle for response is: automate the movement of information and the execution of approved actions; do not automate decisions outside delegated authority. A workflow engine routes a containment request to the right approver with the right context, tracks the clock, escalates when nobody answers, executes the approved action through a scoped control (Chapter 27) and verifies the effect (Chapter 38). The decision itself stays with a human holding authority unless a policy explicitly delegates it to automation for a bounded class of actions. Machine-readable playbook formats such as OASIS CACAO v2.0 can describe such workflows, including steps that require human input.
31.2 Timers, retries and escalation
Each waiting state has a timer. On expiry: escalate to a deputy or higher role; after a second expiry, expire the request with a recorded outcome. Never auto-approve. Retries apply to machine steps with transient failure (an API timeout), with bounded attempts and backoff; human tasks are escalated, not retried. Each retry and escalation is an audit event. Lab 31 uses two 30-minute timers: escalation to the deputy at 30, expiry with no containment at 60. Neither timer ever converts the duty manager's silence into an approval of the containment request.
31.3 Audit trail
For every transition record: time, previous and next state, actor (or "clock"), reason, and references to the request, approval and evidence. The trail should allow a reviewer to answer: who knew what, when, and who decided? Sagas (Garcia-Molina and Salem, 1987) — sequences of steps each with a compensating action — are a useful model for multi-step containment that may need to be unwound.
The Lab 31 trail meets this test for the self-approval scenario. A reviewer can read that analyst-1 requested at t = 0 and tried to self-approve at 5. The clock then escalated the request at 30 and expired it at 60 without containment, so every transition names an actor or the clock and a reason.
31.4 Worked example (synthetic): approval timeout at HVSS
"""Lab 31 (synthetic): case workflow with human approval task, timeout, escalation and audit trail."""
APPROVAL_TIMEOUT = 30 # minutes before escalation
ESCALATION_TIMEOUT = 30 # minutes before the request expires (no automatic approval, ever)
def run(events):
state, t_req, audit = "triage", None, []
def go(t, new, why):
nonlocal state; audit.append((t, state, new, why)); state = new
for t, ev, actor in events + [(10**6, "tick", "clock")]:
# timers fire first, in order
if state == "awaiting_approval" and t_req is not None and t - t_req >= APPROVAL_TIMEOUT:
go(t_req + APPROVAL_TIMEOUT, "escalated", "approver silent 30 min -> deputy paged")
if state == "escalated" and t - t_req >= APPROVAL_TIMEOUT + ESCALATION_TIMEOUT:
go(t_req + APPROVAL_TIMEOUT + ESCALATION_TIMEOUT, "expired", "no approval; containment NOT executed")
if ev == "request" and state == "triage": t_req = t; go(t, "awaiting_approval", f"request by {actor}")
elif ev == "approve" and state in {"awaiting_approval", "escalated"}:
if actor == "analyst-1": audit.append((t, state, state, "REFUSED: requester cannot approve"))
else: go(t, "containing", f"approved by {actor}")
elif ev == "done" and state == "containing": go(t, "closed", "containment verified")
return state, audit
scenarios = {
"approved in time": [(0, "request", "analyst-1"), (12, "approve", "duty-mgr"), (40, "done", "engineer")],
"self-approval": [(0, "request", "analyst-1"), (5, "approve", "analyst-1")],
"deputy approves": [(0, "request", "analyst-1"), (45, "approve", "deputy-mgr"), (70, "done", "engineer")],
"nobody answers": [(0, "request", "analyst-1")],
}
for name, ev in scenarios.items():
final, audit = run(ev)
print(f"[{name}] final={final}")
for a in audit: print(f" t={a[0]:>3} {a[1]} -> {a[2]}: {a[3]}")
Expected output:
[approved in time] final=closed
t= 0 triage -> awaiting_approval: request by analyst-1
t= 12 awaiting_approval -> containing: approved by duty-mgr
t= 40 containing -> closed: containment verified
[self-approval] final=expired
t= 0 triage -> awaiting_approval: request by analyst-1
t= 5 awaiting_approval -> awaiting_approval: REFUSED: requester cannot approve
t= 30 awaiting_approval -> escalated: approver silent 30 min -> deputy paged
t= 60 escalated -> expired: no approval; containment NOT executed
[deputy approves] final=closed
t= 0 triage -> awaiting_approval: request by analyst-1
t= 30 awaiting_approval -> escalated: approver silent 30 min -> deputy paged
t= 45 escalated -> containing: approved by deputy-mgr
t= 70 containing -> closed: containment verified
[nobody answers] final=expired
t= 0 triage -> awaiting_approval: request by analyst-1
t= 30 awaiting_approval -> escalated: approver silent 30 min -> deputy paged
t= 60 escalated -> expired: no approval; containment NOT executed
Four behaviours are visible. In the normal path the approval arrives in 12 minutes. A self-approval is refused and recorded — and the request then follows the timeout path, because refusal is not approval. When the primary approver is silent, the deputy is paged at 30 minutes and approves at 45. When nobody answers, the request expires at 60 minutes with the explicit outcome "containment NOT executed". The engine never treats silence as consent; the expired request remains visible so the duty manager can decide whether the emergency path (Chapter 2) is justified.
31.5 Testing
Test with simulated clocks so timeouts can be exercised in milliseconds. Cases: approval before, at and after timeout boundary; approval by requester; approval by unauthorised role; executor failure after approval (retry then failure state, not silent success); concurrent approval and expiry (the first committed transition wins; the second is rejected and logged). Concurrency bugs in workflow engines are subtle; serialise transitions per case.
The Lab 31 scenarios already cover the requester approving and the silent approver. The boundary cases are still missing: approvals at minutes 29, 30 and 31 relative to the escalation timer, and a deputy approval arriving at exactly 60, when expiry also fires.
31.6 Errors and limits
- Auto-approval on timeout.
- Retrying human tasks instead of escalating.
- Workflow state held only in memory.
- Audit trails that omit clock-driven transitions.
Lab 31 is written to avoid the first and last errors. In the "nobody answers" scenario, the request ends expired with "containment NOT executed" instead of being approved by the clock, and both clock transitions appear in the trail. The approval task is escalated to the deputy rather than resent to the same silent duty manager. The lab still keeps state in a local variable, so a production engine would need to persist it per case to survive a restart mid-wait.
Exercises — chapter 31
EX061 — Boundary. Objective: test edge timing. Inputs: Lab 31. Steps: add scenarios: approval at exactly t = 30 by duty-mgr; approval at t = 60 by deputy-mgr. Report and explain the outcomes. Expected evidence: outputs and explanation. Rubric: 3 = correct outcomes (t = 30: escalation fires first, then approval accepted from escalated state; t = 60: expiry fires first, approval rejected) with discussion of whether boundary semantics are acceptable; 2 = outputs; 1 = attempt; 0 = none.
EX062 — Saga. Objective: design compensation. Inputs: a three-step containment: disable account, revoke sessions, block source address at firewall. Steps: write compensating actions and the behaviour if step 3 fails. Expected evidence: table and behaviour. Rubric: 3 = compensations noted as partial (revocation not undoable), behaviour on failure keeps steps 1–2 and escalates; 2 = partial; 1 = vague; 0 = none.
Chapter synthesis
Orchestrate information and approved actions, not unauthorised decisions. Use explicit state machines, timers that escalate and expire but never approve, bounded retries for machine steps, complete audit trails and tests driven by simulated clocks.
Continue studying — chapter 31 of 50
← Chapter 30 · Course chapter index · Chapter 32 →
Continue to the next authored lesson.