Chapter 2 — Logs and telemetry | Technical Security operations and blue team

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 2 — Logs and telemetry

Lesson map

Lesson 02 published. Map, example, EX005.

flowchart TD
  N0["Learning outcomes"]
  N1["2.1 What logs are"]
  N2["2.2 Key sources"]
  N3["2.3 Useful entries"]
  N4["2.4 Time"]
  N5["2.5 Retention"]
  N6["2.6 Collection"]
  N7["Normalisation"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX005"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["2.1 What logs are"]
  N2["2.2 Key sources"]
  N3["2.3 Useful entries"]
  N4["2.4 Time"]
  N5["2.5 Retention"]
  N6["2.6 Collection"]
  N7["Normalisation"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX005"]

[Learning outcomes]
    |
    v
[2.1 What logs are]
    |
    v
[2.2 Key sources]
    |
    v
[2.3 Useful entries]
    |
    v
[2.4 Time]
    |
    v
[2.5 Retention]
    |
    v
[2.6 Collection]
    |
    v
[Normalisation]
    |
    v
[Worked example] --> [Exercise EX005]

Learning outcomes

You will (1) explain what logs and telemetry are; (2) list key log sources for a SOC; (3) describe what a useful log entry contains; (4) explain log retention and time synchronisation.

2.1 What logs are

A log is a record written by a system when something happens: a logon, a file change, a network connection. Telemetry is the broader stream of such data sent to the SOC. NIST SP 800-92 (Kent and Souppaya, 2006) is the classic guide to log management. For the parks01 case, the portal's authentication log recorded each attempt, the domain controller recorded the account check, and the file share's audit log recorded what was opened. None of these systems was set up for the SOC, so each log answers only the questions its owner chose to record.

2.2 Key sources

For Northgate: the remote-access portal, domain controllers, endpoint agents, the email gateway, the web proxy and the firewall. The 09:04 alert came from the portal's authentication log. Each source answers a different question in the council's cases. The portal and domain controllers show who signed in, the endpoint agents show what ran on a PC, and the email gateway shows who received a message. The proxy shows who clicked, as in the finance case, and the firewall shows traffic that bypassed the proxy.

2.3 Useful entries

A useful entry has a time, source, user, action and outcome. The portal entry "09:09 OK 203.0.113.80 parks01" has all five. Entries missing the user or source are much harder to use. The parks01 entry tells Carlos when it happened (09:09), from where (203.0.113.80), as whom (parks01), what was attempted (a logon) and how it ended (OK). A line with no source address could not have been linked to the 46 failures, and the case would have looked like an ordinary morning logon by a gardener.

2.4 Time

All systems must agree on time; Northgate uses a central time server. If the portal clock were five minutes slow, the logon would seem to come before the guessing, confusing the story. Carlos confirmed the order by comparing two independent sources: the portal and the domain controller both recorded 09:09. If one clock had drifted, the domain controller line would have appeared minutes away from the portal line, and he would have had to correct for the offset before building the timeline.

2.5 Retention

Logs are kept for a set period, here 90 days searchable and one year archived, as agreed with the records manager. Without retention, past activity cannot be checked. Ninety searchable days cover most investigations, including checking whether 203.0.113.80 had tried the portal before Monday. The one-year archive matters for slower questions, such as when the parks01 password was last changed. Agreeing retention with the records manager keeps it lawful as well as useful, since logs contain staff activity. The retention period is written into the logging standard so every new system follows it from the day it is connected.

2.6 Collection

Logs flow into a central security information and event management (SIEM) system where analysts search and correlate them. Having all sources in one SIEM is what lets Carlos see the portal failures, the parks01 logon and the domain controller check side by side. Without central collection he would have had to sign in to each system separately and line up times by hand, which is slow and easy to get wrong during a live case. Normalising fields on arrival is what makes a single search across all six sources possible.

Normalisation

Different systems name the same thing differently: one says user, another account, another principal. The SIEM maps them to common field names when logs arrive, so one search finds a person's activity across all sources.

Worked example

Carlos checks the portal and domain controller logs. The portal shows parks01 logged in at 09:09; the domain controller shows parks01 authenticated at 09:09 from the portal server. Times agree, so he can trust the sequence.

Synthetic case. Log entries are synthetic.

Errors and limits

More logs cost money to store and search; collect what supports decisions.

Exercises

EX005 — Five fields. Name five fields of a useful log entry. Rubric: 3 = time, source, user, action, outcome; 2 = four; 1 = two; 0 = none.
EX006 — Time agreement. Why must clocks agree? Rubric: 3 = otherwise event order is wrong; 2 = partial; 1 = vague; 0 = none.
EX007 — Log sources. List four Northgate log sources. Rubric: 3 = portal, DCs, endpoint agents, email, proxy, firewall (four); 2 = two; 1 = one; 0 = none.
EX008 — Retention purpose. Why keep logs for a set period? Rubric: 3 = to check past activity; 2 = partial; 1 = vague; 0 = none.

Synthesis

Logs with complete fields, agreed time and suitable retention are the raw material of the SOC.

Continue studying — chapter 02 of 18

← Chapter 01 (draft) · Course chapter index · Chapter 03 (draft) →