Chapter 22 — Security operations and blue team: data and dependencies
Lesson map
Lesson 22 published. Map, example, EX064.
flowchart TD
N0["Learning outcomes"]
N1["22.1 Data inventory"]
N2["22.2 Quality"]
N3["22.3 Latency"]
N4["22.4 Retention"]
N5["22.5 External dependencies"]
N6["22.6 Source health"]
N7["22.7 Ownership"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX064"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["22.1 Data inventory"]
N2["22.2 Quality"]
N3["22.3 Latency"]
N4["22.4 Retention"]
N5["22.5 External dependencies"]
N6["22.6 Source health"]
N7["22.7 Ownership"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX064"]
[Learning outcomes]
|
v
[22.1 Data inventory]
|
v
[22.2 Quality]
|
v
[22.3 Latency]
|
v
[22.4 Retention]
|
v
[22.5 External dependencies]
|
v
[22.6 Source health]
|
v
[22.7 Ownership]
|
v
[Worked example] --> [Exercise EX064]
Learning outcomes
You will (1) inventory the SOC's data sources and their owners; (2) assess data quality, latency and retention; (3) identify dependencies on external services; (4) plan for source failures.
22.1 Data inventory
Lucas lists every source: identity sign-ins, mail events, mailbox audit, firewall logs, DNS resolver logs, endpoint logs from staff devices, research server logs (partial), and the phishing report mailbox. For each, he records owner, format, latency, retention and which rules use it.
The rule column proved the most useful. It showed that the outbound-mail check depended on mail events alone, and that no rule used the mailbox audit source at all, even though it recorded the forwarding and deletion rules in the campaign. That gap later led to the correlation rule tested in the first laboratory.
22.2 Quality
Some sources have gaps: DNS logs miss queries from devices using external resolvers; research servers forward inconsistently. Quality problems limit what detections can promise.
Lucas records each gap beside the detections it affects. A rule that counts lookups of a look-alike domain, for instance, can promise only a lower bound because of the DNS gap. Stating this in the rule's description stops analysts from treating a low count as proof that few people clicked. Research server rules carry a similar note about inconsistent forwarding.
22.3 Latency
Cloud mail audit events arrive up to 30 minutes late; sign-ins within 5. Detections that combine them must allow for the slower source.
A rule that looks for a mailbox rule within sixty minutes of a risky sign-in must therefore wait at least thirty minutes after the window closes before deciding nothing happened. Running it too early would miss matches whose audit events had not yet arrived, and an analyst would wrongly conclude the sign-in was harmless. Lucas schedules the correlation to run with this delay built in.
22.4 Retention
Cloud audit logs are kept by the provider for 90 days on Riverton's licence; the SOC copies key events to its own store for longer.
The difference matters for slow attacks. A compromise noticed after three months would find the provider's logs already deleted. The SOC therefore copies sign-ins, rule changes and forwarding events to its own store, which follows the retention rules in the privacy chapter, so key evidence survives longer than the licence allows. The copied events are hashed on arrival, like other evidence.
22.5 External dependencies
The SOC depends on the cloud provider's logging, the identity provider's API and the university network team. A change in any can break collection silently.
The firewall replacement in the worked example below is one such change. Others include a provider renaming a field in its export, which can make a rule stop matching without any error, or an expired API credential. Lucas's inventory lists who to contact for each dependency and how a failure would show itself. Each entry was confirmed with its owner.
22.6 Source health
A daily report shows events received per source compared with the usual range. A sudden drop prompts investigation.
The report compares each source's daily count with the range of the previous four weeks. A drop to zero is obvious, but a fall by half may mean a partial failure, such as one building's controller no longer forwarding. The report goes to the morning shift, which checks any flagged source before relying on it. In its first month, the report caught the firewall gap described below.
22.7 Ownership
Each source has a named owner outside the SOC who is told when problems appear.
Owners are outside the SOC because they control the systems that produce the data. When the research firewall stopped forwarding, the SOC could not fix it; the network team could. Naming the owner in the inventory meant Lucas knew whom to contact within minutes rather than asking around the IT department. Owners also receive the monthly source-health summary, so they can see their system's role in detection and plan changes such as replacements with forwarding in mind. That reduced surprises like the firewall gap.
Cost of sources
Each source costs licence, storage and analyst attention. Lucas adds a column for the number of alerts each source produced last term, so low-value, high-cost sources can be reviewed.
Worked example
Lucas's health report shows the research network firewall sent no logs for two days. The network team had replaced the firewall and not re-enabled forwarding. They fix it within an hour; the gap is recorded so analysts know research traffic for those days is unmonitored.
Synthetic case. Sources and figures are invented.
Errors and limits
Inventories go out of date quickly; review them each term.
Exercises
- EX064 — Inventory fields. What does Lucas record for each source? Rubric: 3 = owner, format, latency, retention, rules using it; 2 = three; 1 = one; 0 = none.
- EX065 — Latency effect. Why must detections combining mail audit and sign-ins allow for delay? Rubric: 3 = mail audit up to 30 min late; 2 = partial; 1 = no reason; 0 = none.
- EX066 — Firewall gap. What caused the research firewall gap, and how was it handled? Rubric: 3 = replacement without forwarding; fixed, gap recorded; 2 = partial; 1 = ignored; 0 = none.
Synthesis
Know every source, its quality and dependencies, and watch source health so gaps are found quickly.
Continue studying — chapter 22 of 40
← Chapter 21 (draft) · Course chapter index · Chapter 23 (draft) →