Chapter 27 — Deployment and validation
Lesson map
Lesson 27 published. Map, example, EX079.
flowchart TD
N0["Learning outcomes"]
N1["27.1 Staged deployment"]
N2["27.2 Validation"]
N3["27.3 Authorisation"]
N4["27.4 Evidence"]
N5["27.5 Failures"]
N6["27.6 Fixes and retest"]
N7["27.7 Exceptions"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX079"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["27.1 Staged deployment"]
N2["27.2 Validation"]
N3["27.3 Authorisation"]
N4["27.4 Evidence"]
N5["27.5 Failures"]
N6["27.6 Fixes and retest"]
N7["27.7 Exceptions"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX079"]
[Learning outcomes]
|
v
[27.1 Staged deployment]
|
v
[27.2 Validation]
|
v
[27.3 Authorisation]
|
v
[27.4 Evidence]
|
v
[27.5 Failures]
|
v
[27.6 Fixes and retest]
|
v
[27.7 Exceptions]
|
v
[Worked example] --> [Exercise EX079]
Learning outcomes
You will (1) deploy detections across many clients safely; (2) validate detections with authorised tests; (3) record validation evidence per client; (4) manage exceptions and failures.
27.1 Staged deployment
New rules go to three pilot clients, then to the rest in groups. Each stage has a check: alert volume within expectations, no errors.
Three pilot clients are enough to show whether a rule floods the queue or throws errors in real data, without exposing all forty clients to a fault at once. For the sequence rule, an expected volume is a handful of alerts a month per client; a pilot client producing dozens a day would stop the rollout until the cause was found.
27.2 Validation
Validation asks whether rules catch the behaviour they target in each client's real environment. Copperleaf runs authorised test actions on each client's systems with a test account: create a local account and add it to Administrators, then remove it.
The test account steps are the second and third parts of the BrightSmile sequence without any of the harm: an account is created, made an administrator and then removed. If the rule fires, Copperleaf knows that this client's servers log the needed events and that they reach the SIEM. If it does not fire, something on that path is broken.
27.3 Authorisation
Tests are listed in the service agreement and scheduled with the client contact. They never touch real data.
Listing the tests in the agreement means the client has already agreed in writing to the kind of action, and scheduling means the contact knows when it will happen. A test account that appeared unannounced on a clinic server would look exactly like support2, and could start a real incident response at the client. The client contact also receives the test account name in advance, so their own staff can recognise it if they notice it.
27.4 Evidence
Each client's validation record shows test time, expected alert, actual alert time and analyst who confirmed.
The difference between expected and actual alert time also shows delay. If the account change happened at 12:00 and the alert arrived at 12:02, the record shows a two-minute path from the client's server to the analyst. Naming the analyst who confirmed means the record can be checked later by asking the person involved. Together the records for all forty clients form the evidence behind the statement that the sequence scenario's coverage is verified rather than assumed.
27.5 Failures
At four clients the sequence rule did not fire. At three, account-management auditing was disabled; at one, the agent was an old version.
Both causes are of the kind chapter 6 warned about: events appear only if auditing enables them, and an agent must be able to forward them. Neither failure showed up in normal operation, because without the test there was simply no alert, and a missing alert looks exactly like a quiet day. Mei records which clients failed and why, so the fixes can be tracked and the same causes checked at future onboardings.
27.6 Fixes and retest
Auditing is enabled and the agent upgraded; retests pass.
The retest repeats the same account test, so a pass shows the fix worked rather than assuming it. For the three clients with auditing disabled, Mei also asks how the onboarding step for audit policy was missed, since a skipped checklist step was the cause at BrightSmile's clinic 2. The upgraded agent at the fourth client is also added to the tool-health list, so its version is watched after future updates. Each retest result is added to the client's validation record beside the failed first attempt, keeping both visible.
27.7 Exceptions
One client's vendor-managed system cannot have auditing enabled; the gap is recorded and the client informed.
Recording the gap means the client's coverage report shows that this system is not covered by the sequence rule, rather than implying it is. The client can then decide whether to ask the vendor for another way to log account changes, or accept the gap knowing what it means. The exception is reviewed at the client's next review, in case the vendor later offers a way to enable auditing on that system. Mei writes the reason for the exception in plain terms, naming the vendor restriction, so a later analyst does not try to enable auditing and breach the vendor's terms.
Scheduling with clients
Tests are booked for quiet periods agreed with each client's contact, often a lunchtime or after hours. The client is told the exact test window and the test account name in advance, so if their own staff notice the account they know it is expected. After the test, Copperleaf confirms by email that the account was removed. Clients who cannot be reached to schedule are listed as unvalidated rather than tested without notice, and the account manager follows up. This keeps validation within what each client has agreed, and it builds trust that Copperleaf's actions on their systems are always announced and reversible.
Worked example
Mei's validation summary: 40 clients tested; 36 passed first time; 3 fixed by enabling auditing; 1 fixed by upgrading the agent; 1 permanent exception recorded for a vendor-managed system at a client that otherwise passed. Coverage for the sequence scenario is now verified rather than assumed.
Synthetic case. Results are invented.
Errors and limits
Validation with test accounts proves the rule fires on that action, not on every variant an attacker might use.
Exercises
- EX079 — Validation test. Describe the authorised validation test. Rubric: 3 = test account created and added to Administrators, then removed; 2 = partial; 1 = attack; 0 = none.
- EX080 — Failure causes. Why did the rule fail at four clients? Rubric: 3 = auditing disabled at 3; old agent at 1; 2 = partial; 1 = unknown; 0 = none.
- EX081 — Exception handling. How was the unfixable system handled? Rubric: 3 = gap recorded; client informed; 2 = partial; 1 = ignored; 0 = none.
Synthesis
Deploy in stages, validate per client with authorised tests and record every pass, fix and exception.
Continue studying — chapter 27 of 30
← Chapter 26 (draft) · Course chapter index · Chapter 28 (draft) →