Chapter 37 — Systems, endpoints and hardware security: operations
Lesson map
Lesson 37 published. Map, example, EX109.
flowchart TD
N0["Learning outcomes"]
N1["37.1 Daily routine"]
N2["37.2 Alerts"]
N3["37.3 Runbooks"]
N4["37.4 Patching"]
N5["37.5 Training"]
N6["Patch management"]
N7["Escalation"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX109"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["37.1 Daily routine"]
N2["37.2 Alerts"]
N3["37.3 Runbooks"]
N4["37.4 Patching"]
N5["37.5 Training"]
N6["Patch management"]
N7["Escalation"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX109"]
[Learning outcomes]
|
v
[37.1 Daily routine]
|
v
[37.2 Alerts]
|
v
[37.3 Runbooks]
|
v
[37.4 Patching]
|
v
[37.5 Training]
|
v
[Patch management]
|
v
[Escalation]
|
v
[Worked example] --> [Exercise EX109]
Learning outcomes
You will (1) define daily operations; (2) set alerts and runbooks; (3) train staff.
37.1 Daily routine
Each morning the technician checks the restore report and repairs failed PCs before opening.
The report lists every PC whose restore failed or did not report, grouped by branch. The technician starts with the branch that opens first and works through the list using the failed-restore runbook. If a PC cannot be repaired before opening, it stays on the out-of-service screen and is swapped later. The routine takes a few minutes on a normal morning, since most nights have zero to two failures.
37.2 Alerts
Per-PC restore failures and the five-failure incident threshold from chapter 7 alert IT.
Per-PC alerts go to the technician's queue for the morning check. The incident alert, fired when five or more PCs fail in one night, goes to Marta directly, because the probability work showed that such a cluster is very unlikely unless something shared has gone wrong. Mateo tested both alerts with synthetic failures on lab PCs before the first operational night. Alerts are tested again after any change to the monitoring system or the contact list.
37.3 Runbooks
Short runbooks cover failed restore, out-of-service PCs and image rollback.
Each runbook fits on one page and starts with how to recognise the problem. Steps are numbered, with the expected result after each, so the person following it knows when something unexpected has happened and should escalate. The runbooks link to each other where needed: a failed restore that keeps failing leads into the out-of-service runbook. Printed copies are kept in each branch. Runbooks are updated after every real use that reveals a missing or unclear step.
37.4 Patching
Monthly OS updates follow SP 800-40r4 planning: test in lab, release to one branch, then all.
The cycle runs on a fixed calendar so service owners know when to expect changes. Millbrook goes first because it is the smallest branch, which limits the effect of any problem. The image's tests and the integration test both run before release. If an update fails at Millbrook, the other branches stay on the previous image, and the failure is fixed in the next build.
37.5 Training
Librarians learn to recognise an out-of-service screen and call IT.
The training takes about twenty minutes and is given at each branch's staff meeting. Librarians see the out-of-service screen on a lab PC, learn which number to call and practise explaining the overnight reset to a patron. They are not asked to repair PCs. Mateo hands out a single card summarising what they need to know, with the dashboard address and the IT phone number. New librarians receive the same session during their first week at a branch.
Patch management
NIST SP 800-40 Rev. 4 describes patch management as preventive maintenance with planning for risk. Fernhill's monthly cycle includes: collect updates, build a test image, run requirement and integration tests, release to Millbrook first, then the other branches after two nights without problems. Emergency updates follow a shorter path with the same tests in compressed form.
Each release's dates and results are recorded, giving evidence of timely patching.
Escalation
If five or more PCs fail in one night, the technician escalates to Marta immediately.
Weekly review
Each week, Marta reviews restore failures, blocks and requests.
Handover between shifts
The technician leaves a short note for the next day's staff.
Measuring operations
Operations measures include restore success rate, time to repair failures and PCs ready at opening, all shown on the dashboard.
Runbook example
The failed restore runbook says: check the dashboard, restart the PC, run manual restore, confirm digests, and if it fails again, swap the PC and log a ticket.
Training records
Each trained person is recorded.
Holiday cover
A backup person covers the technician's leave, using the same runbooks.
Review
Runbooks are reviewed quarterly.
Worked example
On the first operational night, two PCs fail to restore; alerts arrive, and both are repaired by 08:30.
Synthetic case. Synthetic.
Errors and limits
Operations without runbooks depend on memory.
Exercises
- EX109 — Morning task. What does the technician do each morning? Rubric: 3 = check restore report, repair failures; 2 = partial; 1 = vague; 0 = none.
- EX110 — Patching flow. How are monthly updates released? Rubric: 3 = lab, one branch, all; 2 = partial; 1 = vague; 0 = none.
- EX111 — First night. What happened on the first night? Rubric: 3 = two failures, fixed by 08:30; 2 = partial; 1 = vague; 0 = none.
Synthesis
Routine, alerts and runbooks keep the service reliable.
Continue studying — chapter 37 of 50
← Chapter 36 (draft) · Course chapter index · Chapter 38 (draft) →