Chapter 12 — Security operations and blue team: data flows
Lesson map
Lesson 12 published. Map, example, EX034.
flowchart TD
N0["Learning outcomes"]
N1["12.1 The path"]
N2["12.2 Collection points"]
N3["12.3 Delays"]
N4["12.4 Gaps"]
N5["12.5 Monitoring the flow"]
N6["12.6 Documentation"]
N7["12.7 Case"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX034"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["12.1 The path"]
N2["12.2 Collection points"]
N3["12.3 Delays"]
N4["12.4 Gaps"]
N5["12.5 Monitoring the flow"]
N6["12.6 Documentation"]
N7["12.7 Case"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX034"]
[Learning outcomes]
|
v
[12.1 The path]
|
v
[12.2 Collection points]
|
v
[12.3 Delays]
|
v
[12.4 Gaps]
|
v
[12.5 Monitoring the flow]
|
v
[12.6 Documentation]
|
v
[12.7 Case]
|
v
[Worked example] --> [Exercise EX034]
Learning outcomes
You will (1) trace how logs flow from client sites to the SOC; (2) identify collection points, transport and storage; (3) detect gaps and delays in data flows; (4) document flows for each client.
12.1 The path
Events travel from the client's endpoints (agent) and firewall (syslog) over the internet to Copperleaf's collectors, then to the SIEM, where rules run and alerts go to the case system.
For BrightSmile, the server's agent sends Windows events and the firewall sends syslog through the collector box at the clinic. Both arrive at Copperleaf's collectors and then the SIEM, where the new-administrator rule ran. A break anywhere on that path stops alerts, which is why Mei learns the path before trusting what she sees in the SIEM.
12.2 Collection points
Agents send directly over encrypted connections. Firewalls send syslog to a small collector box at each client site, which forwards it securely.
The difference between the two methods is practical rather than a matter of preference. An agent on CLINIC2-SRV can send its own events securely, but a small clinic firewall usually speaks only syslog and cannot encrypt it. The collector box sits inside the clinic, receives that syslog over the local network and forwards it securely, so firewall lines never cross the internet unprotected. Because the collector box has its own power and network connection, Mei lists it in the BrightSmile inventory as a monitored device like any other.
12.3 Delays
Agent events usually arrive within a minute; firewall logs within five. Cloud email audit logs can take up to 30 minutes.
These delays affect how analysts should read recent results in the SIEM. A firewall event at 01:30 may arrive at 01:35, so an empty firewall result at 01:31 does not prove that nothing happened. For a suspected email account takeover, Mei waits for the cloud audit log to catch up before concluding that a mailbox rule was not created. Mei records these typical delays on each client's data-flow diagram, so an analyst can judge how long to wait before treating a missing event as absent.
12.4 Gaps
A collector box that loses power, an expired certificate or a firewall update that resets logging settings can stop data silently.
Each of these failures looks the same in the SIEM: the source goes quiet. A quiet source at night could be a sleeping clinic or a broken feed, which is why the next measure compares volumes against what the source normally sends rather than waiting for someone to notice that alerts have stopped. None of these causes is an attack, yet each leaves the SOC blind in the same way as an attacker who deliberately stopped the logging.
12.5 Monitoring the flow
Copperleaf tracks events per source per hour; a sudden drop to zero raises a data-gap alert to engineering, not to analysts' main queue.
Sending these alerts to engineering keeps analysts focused on security alerts while still making sure someone fixes the feed. Each data-gap alert records which source dropped and when, so if an incident later falls into that window, the case can state plainly that the source was not reporting at the time. Hourly counts also reveal a source that drops gradually, for example a firewall whose log volume halves after an update changed its settings.
12.6 Documentation
Each client has a data-flow diagram: sources, collectors, protocols and expected volumes.
For BrightSmile, the diagram shows the agent on each server and PC, the firewall sending syslog to the collector box at each clinic and the cloud email audit log pulled directly. The expected volume beside each source is what the gap monitor compares against, and the verified date shows when someone last checked the diagram against reality. The diagram is short enough to read during an alert, which matters when an analyst needs to know quickly whether a quiet source is expected.
12.7 Case
During onboarding of a new client, Mei checks the diagram against actual data and finds firewall logs missing.
The diagram said firewall syslog was flowing, but the SIEM showed no firewall events from that client at all. A diagram records intent, while the SIEM records what actually arrives, so Mei treats any disagreement between them as a fault to investigate before she accepts the onboarding as complete. Checking this during onboarding, before the client relies on the service, is far cheaper than discovering the gap during an incident.
Integrity in transit
Logs travel over encrypted, authenticated connections so they cannot be read or altered on the way. Collectors buffer events if the internet link drops and send them when it returns, so short outages delay data rather than lose it. Buffers have limits; long outages still lose events, which the gap report records.
Worked example
The new client's firewall had syslog configured to send to an old address. Mei corrects it with the client's IT contractor, then confirms in the SIEM that firewall events arrive at the expected volume. She updates the diagram's "verified" date.
Synthetic case. Flows and clients are invented.
Errors and limits
Volume checks catch total loss but not partial loss of some event types.
Exercises
- EX034 — Arrival times. Give typical arrival delays for agent, firewall and cloud email logs. Rubric: 3 = ~1 min, ~5 min, up to 30 min; 2 = two; 1 = one; 0 = none.
- EX035 — Silent gaps. Name three causes of silent data gaps. Rubric: 3 = collector power loss, expired certificate, firewall update resetting logging; 2 = two; 1 = one; 0 = none.
- EX036 — Onboarding fix. What was wrong at the new client and how was it confirmed fixed? Rubric: 3 = syslog sent to old address; corrected; volume verified in SIEM; 2 = partial; 1 = vague; 0 = none.
Synthesis
Knowing every data flow, its delay and its failure modes keeps a SOC from monitoring blind.
Continue studying — chapter 12 of 30
← Chapter 11 (draft) · Course chapter index · Chapter 13 (draft) →