Chapter 37 — Network and communications security: operations | Engineering Network and communications security

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 37 — Network and communications security: operations

Lesson map

Lesson 37 published. Map, example, EX109.

flowchart TD
  N0["Learning outcomes"]
  N1["37.1 Daily and weekly routines"]
  N2["37.2 Change management"]
  N3["37.3 Monitoring health"]
  N4["37.4 On-call and escalation"]
  N5["Learning from incidents in operation"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX109"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["37.1 Daily and weekly routines"]
  N2["37.2 Change management"]
  N3["37.3 Monitoring health"]
  N4["37.4 On-call and escalation"]
  N5["Learning from incidents in operation"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX109"]

[Learning outcomes]
    |
    v
[37.1 Daily and weekly routines]
    |
    v
[37.2 Change management]
    |
    v
[37.3 Monitoring health]
    |
    v
[37.4 On-call and escalation]
    |
    v
[Learning from incidents in operation]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX109]

Learning outcomes

You will (1) define operational routines; (2) manage changes; (3) monitor health.

37.1 Daily and weekly routines

Operations depend on small routines done reliably. Each morning, an engineer checks tunnel status, firewall pair health and overnight alerts, which takes about ten minutes. Weekly, the team reviews unexpected denies by zone pair, certificates expiring within sixty days and configuration drift reports from the comparison script. Monthly, they review rule usage and remove unused rules after confirming with each rule's owner. Every routine has a written checklist, an expected duration and a place to record results, so anyone can see whether checks were done and what they found.

37.2 Change management

Policy changes follow the process designed earlier: a request stating the reason and owner, review against the needed-flows list, a lab test against the regression suite, a scheduled deployment and verification afterwards. Emergency changes are permitted when a service is down, but they must be recorded within one working day and reviewed at the next weekly meeting, so urgency never becomes a permanent bypass. Kofi tracks the change failure rate, meaning the share of changes that cause an incident or rollback, because it shows whether the process is actually preventing mistakes.

37.3 Monitoring health

Dashboards show tunnel availability, deny trends by zone pair, firewall processor and memory use, and the rate at which logs arrive in the central store. That last measure is easy to overlook but important. If a firewall stops sending logs, the dashboards simply go quiet, which looks exactly like a calm network. Kofi therefore alerts when log arrival from any device drops well below its normal rate. A silent logging failure would blind every other control that depends on logs, so monitoring the monitoring is treated as a core task rather than an extra.

37.4 On-call and escalation

Out of hours, only two kinds of alert page the on-call engineer: loss of tunnels at offices that run critical services, and failure of the main firewall pair. Everything else waits for the next working day, when it is handled through the normal routines. Kofi keeps the paging list short deliberately, because engineers who are woken often by alerts that turn out to be harmless learn to ignore pages, and an ignored page during a real outage is worse than no page at all. The list is reviewed quarterly against actual pages received.

Learning from incidents in operation

When something does go wrong in operation, the council engineers hold a short blameless review within a week. The review records what happened, how it was detected, how long detection and recovery took, and which routine or control would have caught it sooner. The rollback caused by a misordered rule led to a new check in the change template: every proposed rule is tested in the lab against the full regression suite, not only against the flow it was meant to allow. Reviews focus on improving the system rather than blaming a person, which encourages engineers to report near misses honestly.

Worked example

In the first three months of operation, the change failure rate is 4 percent, with one rollback caused by a rule placed in the wrong order.

Synthetic case. Figures are synthetic.

Errors and limits

Routines that rely on one engineer fail during leave; train both engineers and document routines.

Exercises

EX109 — Weekly items. List three weekly operational checks. Rubric: 3 = three;; 2 = two;; 1 = one;; 0 = none.
EX110 — Silent logging. Why alert on a drop in log arrival? Rubric: 3 = logging failure blinds monitoring;; 2 = visibility;; 1 = vague;; 0 = none.
EX111 — Paging policy. Why page only for critical alerts out of hours? Rubric: 3 = keep pages meaningful;; 2 = fatigue;; 1 = vague;; 0 = none.

Synthesis

Simple, recorded routines and change management keep the segmented network healthy.

Continue studying — chapter 37 of 50

← Chapter 36 (draft) · Course chapter index · Chapter 38 (draft) →