OpenCyber Academy
Technologist · Chapter 19 · English
Course code: OCA-080Chapter 19: OCA-080-C019

Chapter 19 — Network and communications security: monitoring

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 19 published. Map, example, EX055.

flowchart TD
  N0["Learning outcomes"]
  N1["19.1 What to monitor"]
  N2["19.2 Alert quality"]
  N3["Lab — Alert precision"]
  N4["19.3 Reading the precision"]
  N5["19.4 Tuning safely"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX055"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["19.1 What to monitor"]
  N2["19.2 Alert quality"]
  N3["Lab — Alert precision"]
  N4["19.3 Reading the precision"]
  N5["19.4 Tuning safely"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX055"]

[Learning outcomes]
    |
    v
[19.1 What to monitor]
    |
    v
[19.2 Alert quality]
    |
    v
[Lab — Alert precision]
    |
    v
[19.3 Reading the precision]
    |
    v
[19.4 Tuning safely]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX055]

Learning outcomes

You will (1) collect and review alerts; (2) measure alert precision; (3) tune detection.

19.1 What to monitor

With central logs in place, Tomás and Gareth decide what to watch. They choose repeated denied connections from student or guest devices towards administration or management, configuration changes on network devices, failed administrator logins, and alerts from the firewall's intrusion detection feature. NIST SP 800-94 describes intrusion detection and prevention systems and their tuning. Monitoring is chosen to match the risks and scenarios from earlier chapters, rather than enabling every possible alert. Each alert type has an owner, a short response note and a review date, so alerts lead to action rather than accumulating unread.

19.2 Alert quality

Not all alerts are useful. Gareth labels a synthetic sample of alerts from the first weeks as true, meaning they needed attention, or false, meaning they were harmless. Tomás calculates precision for each rule: the share of its alerts that were true. Rules with low precision waste Gareth's limited time and teach him to ignore alerts, which could cause a real one to be missed. Measuring precision turns a vague feeling that "there are too many alerts" into evidence about which rules need tuning.

Lab — Alert precision

"""Alert volume and precision by rule from a synthetic, labelled IDS sample (stdlib only)."""
import collections
alerts = [("policy-p2p", True)] * 4 + [("policy-p2p", False)] * 2 + [("dns-tunnel-heuristic", False)] * 9 + \
         [("dns-tunnel-heuristic", True)] * 1 + [("old-tls", True)] * 6 + [("smb-from-guest", True)] * 3
tot, tp = collections.Counter(), collections.Counter()
for r, ok in alerts: tot[r] += 1; tp[r] += ok
for r in tot: print(f"{r:22}{tot[r]:3} alerts  precision {tp[r]/tot[r]:.2f}")
print(f"overall {sum(tp.values())}/{len(alerts)} = {sum(tp.values())/len(alerts):.2f}")

Run with python3 lab_alerts.py. Verified output:

policy-p2p              6 alerts  precision 0.67
dns-tunnel-heuristic   10 alerts  precision 0.10
old-tls                 6 alerts  precision 1.00
smb-from-guest          3 alerts  precision 1.00
overall 14/25 = 0.56

19.3 Reading the precision

Of twenty-five alerts, fourteen were true, an overall precision of 0.56. The DNS tunnelling heuristic produced ten alerts with only one true, precision 0.10, mostly triggered by the learning platform's long, random-looking names. The old TLS rule and the guest file-sharing rule had precision 1.00. The peer-to-peer policy rule had 0.67. Tomás recommends tuning the DNS heuristic by excluding the learning platform's known domains, which should remove most false alerts while keeping the rule active for other traffic. High-precision rules are left unchanged.

19.4 Tuning safely

Tuning must not hide real problems. Tomás makes the DNS exclusion as narrow as possible, naming specific domains rather than whole categories, records why it was added and sets a review date. After tuning, he checks the next two weeks: DNS alerts fall from about ten to two a fortnight, and both are worth a look. He also keeps a small set of test events that should always alert, so he can confirm tuning has not accidentally silenced the rule entirely.

Worked example

Overall precision is 0.56; the DNS heuristic's 0.10 precision is improved by a narrow, documented exclusion.

Synthetic case. Alerts and labels are synthetic.

Errors and limits

Broad exclusions added to reduce noise can silence real detections.

Exercises

Synthesis

Measuring alert precision and tuning narrowly keep monitoring useful for a small team.

Continue studying — chapter 19 of 30

← Chapter 18 (draft) · Course chapter index · Chapter 20 (draft) →

Worked answers

EX055. Fourteen of twenty-five, precision 0.56.

EX056. The learning platform's long, random-looking names triggered it.

EX057. He keeps test events that should always trigger the rule.