Chapter 38 — Security operations and blue team: validation | Engineering Security operations and blue team

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 38 — Security operations and blue team: validation

Lesson map

Lesson 38 published. Map, example, EX112.

flowchart TD
  N0["Learning outcomes"]
  N1["38.1 Validation questions"]
  N2["38.2 Purple-team validation"]
  N3["38.3 Technique selection"]
  N4["38.4 Outcomes"]
  N5["38.5 Real incidents"]
  N6["38.6 Avoiding gaming"]
  N7["38.7 Repeat"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX112"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["38.1 Validation questions"]
  N2["38.2 Purple-team validation"]
  N3["38.3 Technique selection"]
  N4["38.4 Outcomes"]
  N5["38.5 Real incidents"]
  N6["38.6 Avoiding gaming"]
  N7["38.7 Repeat"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX112"]

[Learning outcomes]
    |
    v
[38.1 Validation questions]
    |
    v
[38.2 Purple-team validation]
    |
    v
[38.3 Technique selection]
    |
    v
[38.4 Outcomes]
    |
    v
[38.5 Real incidents]
    |
    v
[38.6 Avoiding gaming]
    |
    v
[38.7 Repeat]
    |
    v
[Worked example] --> [Exercise EX112]

Learning outcomes

You will (1) validate that detections work against realistic behaviour; (2) plan authorised purple-team validation; (3) measure detection coverage of priority techniques; (4) feed validation results into improvement.

38.1 Validation questions

Validation asks whether the SOC detects what matters in real conditions. Verification (chapter 30) proved rules match their specifications; validation tests whether the specifications and the whole system catch realistic behaviour.

A rule can match its specification perfectly while the specification itself is wrong, for example by assuming attackers always fail many times before succeeding. Validation tests the assumptions by using realistic behaviour, which may differ from what engineers imagined. It also checks the whole system, including analysts and playbooks, rather than rules alone.

38.2 Purple-team validation

In a purple-team exercise, an authorised team performs agreed, safe simulations of techniques while defenders watch. Scope, timing and stop conditions are approved by the CISO. The SOC sees which simulations produced alerts, which produced only logs and which left nothing.

Watching together is what makes it purple rather than red: the goal is learning, not catching the defenders out. Stop conditions protect operations, such as halting if a simulation affects a live service. Recording which simulations left no trace at all identifies data gaps, often the most valuable finding of the exercise.

38.3 Technique selection

Lindqvist chooses twelve priority ATT&CK techniques from its threat analysis, including valid account use, new services and remote logons into depot networks.

Twelve is enough to cover the most important risks without overwhelming the team in one exercise. Choosing from the threat analysis keeps the selection relevant to Lindqvist rather than generic. Including depot techniques tests the newly built monitoring, which has not yet seen realistic activity. The list is agreed with the CISO, and each technique is linked to the threat analysis entry that justified it, so the selection can be revisited when the analysis changes.

38.4 Outcomes

For each simulation the result is: alerted, logged but no alert, or not logged. Each category leads to a different fix: tuning, new detection or new data source.

Alerted means the full chain worked. Logged but no alert means the data exist but no rule uses them, so the fix is a new or tuned detection. Not logged means the SOC was blind, so the fix starts with collecting a new source. Counting outcomes across techniques gives a clear coverage picture.

38.5 Real incidents

Validation also uses real incidents: did the SOC detect them, or did customers report them first? The portal takeover was detected internally — a validation point for R-07.

Real incidents test the system against actual attackers rather than simulations. A customer reporting an incident first is a sign that detection failed somewhere. The portal takeover being caught internally supports R-07, though a single case is limited evidence, and the SOC keeps counting over time. Amara logs each incident's detection source in the case system so the share found internally can be counted.

38.6 Avoiding gaming

Defenders do not get the simulation script in advance; otherwise detections are tuned to the test rather than the behaviour.

If defenders know exactly which simulations will run and when, they can tune rules to catch those specific events, which inflates results without improving real detection. Keeping the details secret, while sharing the general scope, ensures the exercise tests how the SOC would perform against unannounced activity. The authorised team keeps its detailed plan with the CISO's office, and defenders learn the specifics only during the debrief after the exercise ends.

38.7 Repeat

Validation repeats quarterly; results feed the coverage metric from chapter 29.

Quarterly repetition shows whether fixes worked and whether coverage drifts as systems change. Techniques that alerted last time are tested again, because a parser change or rule edit could have broken them. The trend over several quarters gives managers a clear view of progress. Results are compared quarter by quarter in a single table, with each technique's outcome over time. A technique that moves from alerted to logged-only is treated as a regression and investigated like a defect, with an owner and a deadline.

Safety of exercises

Simulations avoid depot systems during dispatch peaks and stop immediately if any business service slows.

Worked example

Of twelve simulated techniques, seven alerted, three were logged but did not alert, and two left no trace — both in depot networks where only denied connections are logged. The team adds an alert for a logged-only technique and raises a project to log allowed connections from depots to domain controllers.

Synthetic case. The exercise and results are invented.

Errors and limits

Simulations are approximations of real attackers and run under known conditions.

Exercises

EX112 — Three outcomes. Name the three simulation outcome categories and their fixes. Rubric: 3 = alerted, logged-no-alert (new rule), not logged (new data source); 2 = two; 1 = one; 0 = none.
EX113 — Exercise results. Summarise the validation results. Rubric: 3 = 7 alerted, 3 logged only, 2 no trace in depots; 2 = partial; 1 = wrong; 0 = none.
EX114 — No script. Why are defenders not given the simulation script? Rubric: 3 = prevents tuning to the test instead of behaviour; 2 = partial; 1 = secrecy; 0 = none.

Synthesis

Validation tests the whole system against realistic behaviour; its results turn directly into tuning, detections and data sources.

Continue studying — chapter 38 of 50

← Chapter 37 (draft) · Course chapter index · Chapter 39 (draft) →