Chapter 20 — Security operations and blue team: troubleshooting | Technologist Security operations and blue team

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 20 published. Map, example, EX058.

flowchart TD
  N0["Learning outcomes"]
  N1["20.1 Systematic diagnosis"]
  N2["20.2 Heartbeats"]
  N3["Lab — heartbeat check"]
  N4["20.3 Reading the result"]
  N5["20.4 Failure or tampering"]
  N6["20.5 Fixes"]
  N7["20.6 Recording"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX058"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["20.1 Systematic diagnosis"]
  N2["20.2 Heartbeats"]
  N3["Lab — heartbeat check"]
  N4["20.3 Reading the result"]
  N5["20.4 Failure or tampering"]
  N6["20.5 Fixes"]
  N7["20.6 Recording"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX058"]

[Learning outcomes]
    |
    v
[20.1 Systematic diagnosis]
    |
    v
[20.2 Heartbeats]
    |
    v
[Lab — heartbeat check]
    |
    v
[20.3 Reading the result]
    |
    v
[20.4 Failure or tampering]
    |
    v
[20.5 Fixes]
    |
    v
[20.6 Recording]
    |
    v
[Worked example] --> [Exercise EX058]

Learning outcomes

You will (1) diagnose monitoring failures systematically; (2) use heartbeats and volumes to locate problems; (3) distinguish failure from tampering; (4) record fixes for future use.

20.1 Systematic diagnosis

When data stops, analysts work along the data path: is the host up, is the agent running, is the network reaching Copperleaf, is the SIEM ingesting? Each check narrows the cause.

The order follows the path the events travel, so each check rules out one part. If the host is up but the agent is not running, the fault is on the host; if the agent runs but nothing reaches the collectors, the network is suspect. Checking in a fixed order prevents an analyst from reinstalling an agent when the real cause was an expired certificate.

20.2 Heartbeats

Agents send a heartbeat every five minutes. Gaps over ten minutes are stale; over an hour, missing.

With a five-minute interval, a ten-minute gap means two heartbeats in a row were missed, which is more than ordinary delay. A gap of over an hour means about twelve heartbeats in a row were missed. The two thresholds separate a brief interruption, such as a clinic's internet blip, from a sustained loss that needs someone to act. Both thresholds appear in the lab code, gap <= 10 for OK and gap <= 60 for stale, so the states printed are exactly the rule described here.

Lab — heartbeat check

"""Find endpoint agents that missed heartbeats (synthetic; expected every 5 minutes)."""
now = 14 * 60
last = {"CLINIC1-SRV": now - 3, "CLINIC1-PC1": now - 4, "CLINIC2-SRV": now - 95, "CLINIC2-PC1": now - 2, "CLINIC3-SRV": now - 18}
for host, t in sorted(last.items()):
    gap = now - t
    state = "OK" if gap <= 10 else ("STALE" if gap <= 60 else "MISSING")
    print(f"{host:12} last heartbeat {gap:3} min ago {state}")

Run with python3 lab_heartbeat.py. Verified output:

CLINIC1-PC1  last heartbeat   4 min ago OK
CLINIC1-SRV  last heartbeat   3 min ago OK
CLINIC2-PC1  last heartbeat   2 min ago OK
CLINIC2-SRV  last heartbeat  95 min ago MISSING
CLINIC3-SRV  last heartbeat  18 min ago STALE

20.3 Reading the result

CLINIC2-SRV last reported 95 minutes ago (missing) and CLINIC3-SRV 18 minutes ago (stale); the others are fine. Two servers need attention, the missing one first.

Mei checks the gaps against the thresholds: 95 minutes is above 60, so CLINIC2-SRV is missing; 18 is above 10 but below 60, so CLINIC3-SRV is stale; and 2, 3 and 4 minutes are all within 10. The order of attention follows from criticality and length of gap, which here both point to CLINIC2-SRV first. Because the lab measures minutes from the current time, the same script run later would print larger gaps for the same hosts.

20.4 Failure or tampering

An attacker might stop the agent. Analysts check what happened just before the gap: was there a logon, a service stop, a reboot? A gap right after a suspicious logon is treated as an incident.

The BrightSmile incident makes this question concrete: an attacker with administrator rights on a server can stop the agent before encrypting files. If a missing heartbeat on CLINIC2-SRV had followed a night logon by reception, Mei would have treated it as part of the incident rather than as a maintenance fault.

20.5 Fixes

CLINIC2-SRV was rebooted for updates and its agent failed to start because of a corrupted update; CLINIC3-SRV's clinic had a brief internet outage. The agent is reinstalled; the outage resolves itself.

The two causes found here call for quite different actions from Copperleaf. A corrupted agent on CLINIC2-SRV needs the contractor to reinstall it, while the internet outage at clinic 3 needs nothing from Copperleaf except to confirm that buffered events arrive once the link returns. Mei checks that CLINIC3-SRV's heartbeat returns to OK before closing that item.

20.6 Recording

Each fix goes into the knowledge base: symptoms, cause, solution.

For CLINIC2-SRV the entry reads, in effect: missing heartbeat after a patch reboot, agent failed to start because of a corrupted update, fixed by reinstalling. The next analyst who sees a missing heartbeat after patching can start from that entry rather than from the beginning of the data path. Entries are short and searchable by symptom, because an analyst at night searches for what they see, not for the cause they have yet to find. After the CLINIC2-SRV fix, the patch system also checks agent health after updates, which the entry records as the prevention step.

Escalating to engineering

Analysts handle simple cases; problems needing collector or SIEM changes go to Copperleaf's engineer with the diagnosis so far, saving repeated checks.

Worked example

Before calling the client about CLINIC2-SRV, Mei checks the last events received: an update installation and a reboot request from the patch system at 12:24, a minute before its last heartbeat at 12:25, and nothing suspicious. She records "probable failed agent start after update" and calls the IT contractor, who confirms and reinstalls the agent.

Synthetic case. Hosts and times come from the lab script and invented follow-up.

Errors and limits

Heartbeats show the agent is alive, not that every event type is being collected.

Exercises

EX058 — Lab states. Which hosts are missing or stale?
Show marking criteria
Rubric: 3 = CLINIC2-SRV missing (95), CLINIC3-SRV stale (18); 2 = one; 1 = wrong; 0 = none.
EX059 — Tampering check. How do analysts tell failure from tampering?
Show marking criteria
Rubric: 3 = look at events just before the gap: logon, service stop, reboot; 2 = partial; 1 = assume failure; 0 = none.
EX060 — Root causes. What caused the two problems?
Show marking criteria
Rubric: 3 = failed agent start after update; brief internet outage; 2 = one; 1 = vague; 0 = none.

Synthesis

Troubleshoot along the data path, check for tampering before assuming failure, and record every fix.

Continue studying — chapter 20 of 30

← Chapter 19 (draft) · Course chapter index · Chapter 21 (draft) →