Chapter 37 — Operating a response system with auditability
Lesson map
Lesson 37 published. Map, example, EX073.
flowchart TD
N0["Learning outcomes and connection"]
N1["37.1 The platform needs its own operations"]
N2["37.2 Change records"]
N3["37.3 Reconciliation"]
N4["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4
N4 --> EX["Worked example"]
EX --> Q["Exercise EX073"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["37.1 The platform needs its own operations"]
N2["37.2 Change records"]
N3["37.3 Reconciliation"]
N4["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX073"]
[Learning outcomes and connection]
|
v
[37.1 The platform needs its own operations]
|
v
[37.2 Change records]
|
v
[37.3 Reconciliation]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX073]
Learning outcomes and connection
After this chapter you can (1) write runbooks and operator roles for a response platform; (2) define service metrics and alerts for the platform itself; (3) run change management that keeps an auditable record; and (4) detect and investigate an unauthorised configuration change. It builds on Chapters 14, 16, 21 and 31 and prepares Chapters 39 and 48.
37.1 The platform needs its own operations
A response platform is a production service with on-call staff, releases, incidents and audits. Its operations plan names: operator roles (platform on-call, release manager, evidence custodian, access administrator — separated where feasible); runbooks for routine and failure tasks (ingest stall, broker failover, certificate renewal, spool replay); service metrics tied to the objectives of Chapter 21; and change management. The SRE practice of error budgets — allowing change while objectives are met, slowing change when they are not — fits well (Beyer et al., Site Reliability Engineering).
37.2 Change records
Every change to detection rules, scoring, workflow definitions, integration permissions or retention settings is recorded: what changed (diff), who requested, who approved, when deployed, test evidence and rollback. Changes are made through a pipeline from version control, not by editing production consoles. The audit log of the platform records configuration changes independently of the change record, so that the two can be reconciled. NIST SP 800-53 Rev. 5 control families for configuration management (CM) and audit and accountability (AU) give a checklist.
37.3 Reconciliation
Daily: compare platform audit events of type "configuration change" with approved change records. Any change without a record is an incident about the platform — possibly an emergency fix not documented, possibly an attacker suppressing detection. Weekly: compare integration permissions in each provider against the declared scopes (Chapter 34).
At HVSS, the daily comparison is what caught the R-118 exception added at 23:47 on 13 May: an audit event of type configuration change with no matching change record. Without it, the exception for partner-mail.example would have stayed in place well beyond its nine hours.
37.4 Worked example (synthetic): an unauthorised change at HVSS
Reconciliation on 14 May flagged an audit event: detection rule R-118 suspicious inbox forwarding changed at 23:47 on 13 May by ops-admin-2, adding an exception for domain partner-mail.example. No change record existed.
Investigation steps: (1) preserve the audit events and the rule's version history; (2) identify the human behind ops-admin-2 — a shared break-glass-style account, so the platform's own access log and the IdP sign-in were needed: sign-in from an HVSS VPN address assigned to an on-call engineer; (3) interview: the engineer had added the exception during a night-time flood of false positives from a newly onboarded partner, intending to file a change in the morning; (4) assess impact: during the 9 hours of the exception, two forwarding rules to that domain were created — both legitimate on review; (5) outcome: emergency change retroactively recorded; shared ops-admin-2 replaced by named accounts with just-in-time elevation; emergency change procedure clarified (record within 1 hour, auto-expiry of emergency exceptions after 24 hours).
The case was benign, but the engineering conclusion is the same as for a malicious one: the platform must make undocumented changes impossible or immediately visible.
37.5 Errors and limits
- Console edits in production.
- Shared operator accounts.
- Change records without independent reconciliation.
- Emergency exceptions without expiry.
The R-118 case from 13 May contains three of these errors at once. The exception was a production console edit made under the shared ops-admin-2 account, so only the VPN address and IdP sign-in identified the engineer. It had no expiry, and only the daily reconciliation against the independent audit log brought it to light. HVSS's fixes map one-to-one: named accounts with just-in-time elevation, emergency changes recorded within an hour, and a 24-hour auto-expiry.
Exercises — chapter 37
EX073 — Operations plan. Objective: write an operations plan extract. Inputs: HVSS platform with rotating on-call and monthly releases. Steps: define four operator roles with separation rules, three platform metrics with alert thresholds and two runbook titles with triggers. Expected evidence: plan. Rubric: 3 = separation explicit, metrics tied to objectives, runbooks with triggers; 2 = partial; 1 = vague; 0 = none.
EX074 — Reconciliation query. Objective: implement reconciliation. Inputs: two synthetic CSV lists: audit config-change events (id, time, actor, object) and change records (change_id, object, approved_window_start, end). Steps: write Python that flags audit events with no change record whose window covers the event time for the same object. Expected evidence: code and test data with one flagged event. Rubric: 3 = correct matching on object and window, edge cases; 2 = object only; 1 = attempt; 0 = none.
Chapter synthesis
Operate the response platform as a production service with separated roles, runbooks, metrics tied to objectives and change management from version control, reconciled daily against an independent audit log so that unauthorised changes are found quickly.
Continue studying — chapter 37 of 50
← Chapter 36 · Course chapter index · Chapter 38 →
Continue to the next authored lesson.