Chapter 33 — Network and communications security: safety and resilience
Lesson map
Lesson 33 published. Map, example, EX097.
flowchart TD
N0["Learning outcomes"]
N1["33.1 Critical services"]
N2["33.2 Degraded modes"]
N3["33.3 Recovery"]
N4["33.4 Exercises"]
N5["Recovery documentation"]
N6["Spare equipment"]
N7["Resilience against misconfiguration"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX097"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["33.1 Critical services"]
N2["33.2 Degraded modes"]
N3["33.3 Recovery"]
N4["33.4 Exercises"]
N5["Recovery documentation"]
N6["Spare equipment"]
N7["Resilience against misconfiguration"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX097"]
[Learning outcomes]
|
v
[33.1 Critical services]
|
v
[33.2 Degraded modes]
|
v
[33.3 Recovery]
|
v
[33.4 Exercises]
|
v
[Recovery documentation]
|
v
[Spare equipment]
|
v
[Resilience against misconfiguration]
|
v
[Worked example] --> [Exercise EX097]
Learning outcomes
You will (1) protect critical services; (2) plan degraded modes; (3) exercise recovery.
33.1 Critical services
Out-of-hours social care and CCTV are critical. They get priority on redundant links.
Critical services are those the council cannot do without for long, such as emergency contact systems, payments and core file access. Listing them with their dependencies, including network zones, links and authentication, shows what must be protected and restored first after a disruption. Each service has an owner who agrees the acceptable outage time.
This list guides both design and recovery planning, so it is reviewed annually.
33.2 Degraded modes
If the main office is isolated, branches keep local services and phones fall back to mobile.
A degraded mode is a planned way of continuing with reduced function, such as an office using a backup internet link with limited services, or staff working from paper forms. Planning degraded modes in advance means a failure leads to a known, safe fallback instead of improvisation, and that security controls still apply during the fallback.
Staff practise switching to degraded modes so they know how.
33.3 Recovery
Configuration backups allow rebuild of a failed firewall within four hours.
Recovery restores normal service after a disruption. Plans list steps in order, starting with critical services, and include restoring configuration from version control, verifying rules, and checking that monitoring works again. Each step has an owner and expected time. Recovery is not complete until services are verified and the cause is understood, so the same failure is less likely to recur.
Lessons are recorded and the plan updated.
Timings are compared with targets.
33.4 Exercises
A tabletop exercise walks through main office loss.
Recovery plans are tested through exercises. A tabletop exercise walks through a scenario on paper, such as a firewall failure at the main office, with each participant describing their actions. A practical exercise restores a configuration in the lab and times it. Exercises reveal missing steps, outdated contacts and unrealistic timings. Findings are recorded, plans updated, and exercises repeated, so the organisation is ready when a real disruption happens.
Exercises should be blameless and realistic.
Recovery documentation
Recovery depends on documentation that is available when the network is not. Kofi keeps printed copies of the recovery runbook, firewall backups on encrypted storage at two sites and contact lists for providers and suppliers. The runbook gives step-by-step instructions for replacing a failed branch firewall from a spare, including which configuration file to load and which tests to run afterwards. He tests the runbook by asking an engineer unfamiliar with the project to follow it in the lab. Gaps found in that test are corrected before the runbook is approved.
Spare equipment
Hardware failures take time to resolve under support contracts. The council keeps one spare branch firewall, preloaded with the common template, at the main office. With the runbook and backups, a failed branch firewall can be replaced within four hours, well inside the time a branch can operate in degraded mode. The spare is updated with firmware quarterly so it is ready when needed. The spare's firmware and template version are logged so engineers know its state before using it. Its location and access arrangements are listed in the runbook so it can be collected quickly out of hours.
Resilience against misconfiguration
Many outages are caused by configuration changes rather than hardware. The design's resilience therefore includes the change process: review, lab testing, rollback plans and change windows. Kofi counts change-related incidents before and after the project as a resilience measure, alongside link availability, reflecting the reality that people and process failures are as important as equipment failures. He also reviews each change-related incident to see which safeguard failed, and strengthens that step. Over the project, this shows change failures falling as the process matures, giving the council evidence that discipline in change management improves availability as much as extra hardware does.
Worked example
The exercise reveals that the out-of-hours line depends on the main office; a backup route is added.
Synthetic case. The exercise is synthetic.
Errors and limits
Resilience untested in exercises often fails in reality.
Exercises
- EX097 — Degraded mode. Describe one degraded mode. Rubric: 3 = branches local, phones to mobile;; 2 = one;; 1 = vague;; 0 = none.
- EX098 — Exercise finding. What did the exercise reveal? Rubric: 3 = out-of-hours line dependency;; 2 = dependency;; 1 = vague;; 0 = none.
- EX099 — Rebuild target. What is the firewall rebuild target? Rubric: 3 = four hours from backups;; 2 = hours;; 1 = vague;; 0 = none.
Synthesis
Planning degraded modes protects critical services.
Continue studying — chapter 33 of 50
← Chapter 32 (draft) · Course chapter index · Chapter 34 (draft) →