Chapter 28 — Systems, endpoints and hardware security: failure analysis
Lesson map
Lesson 28 published. Map, example, EX082.
flowchart TD
N0["Learning outcomes"]
N1["28.1 Data"]
N2["Lab — update failures"]
N3["28.2 Reading the result"]
N4["28.3 Fixes"]
N5["28.4 Re-measure"]
N6["Pending reboots"]
N7["Unknown failures"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX082"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["28.1 Data"]
N2["Lab — update failures"]
N3["28.2 Reading the result"]
N4["28.3 Fixes"]
N5["28.4 Re-measure"]
N6["Pending reboots"]
N7["Unknown failures"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX082"]
[Learning outcomes]
|
v
[28.1 Data]
|
v
[Lab — update failures]
|
v
[28.2 Reading the result]
|
v
[28.3 Fixes]
|
v
[28.4 Re-measure]
|
v
[Pending reboots]
|
v
[Unknown failures]
|
v
[Worked example] --> [Exercise EX082]
Learning outcomes
You will (1) analyse update failures by cause; (2) find the biggest fixable causes; (3) reduce failures.
28.1 Data
Failure codes from a month of update attempts.
The management server records an error code each time an update fails to install. Aisha exports one month of these codes for all laptops and groups them by cause. A laptop that fails twice in the month counts twice, because each failure needs a restart, a retry or a fix. Grouping by code turns 34 separate error lines into five causes, which is short enough for Gareth to act on in one sitting.
Lab — update failures
"""Group synthetic update failure codes by cause (stdlib only)."""
import collections
codes = ["disk_full"]*9 + ["pending_reboot"]*14 + ["offline"]*6 + ["driver_conflict"]*3 + ["unknown"]*2
c = collections.Counter(codes)
for k, v in c.most_common(): print(f"{k:16} {v:2d} ({v/len(codes)*100:.0f}%)")
print(f"total {len(codes)}")
Run with python3 lab_failures.py. Verified output:
pending_reboot 14 (41%)
disk_full 9 (26%)
offline 6 (18%)
driver_conflict 3 (9%)
unknown 2 (6%)
total 34
28.2 Reading the result
Of 34 failures, pending reboots cause 14 (41%), full disks 9 (26%), offline laptops 6 (18%), driver conflicts 3 (9%) and unknown 2 (6%).
Pending reboots and full disks together account for 23 of the 34 failures, about two thirds. Both are operational causes that Bramley can address with policy and housekeeping, not with a new tool. Only the two unknown failures need individual investigation, and one turns out to be a corrupted update cache. Offline laptops, six failures, are the third cause and overlap with the home workers already seen in the patch tail.
28.3 Fixes
Forced restarts address reboots; disk clean-up tasks address full disks; VPN-based updates address offline laptops.
The forced restart after seven days' notice, with a reminder on day five, targets the 14 pending reboots. A scheduled clean-up removes temporary files, and two laptops get larger disks. Internet-based management, already tried on five home workers, removes the dependence on the VPN for the six offline laptops. Driver conflicts, three failures, are reported to the laptop vendor, because Bramley cannot fix vendor drivers itself.
28.4 Re-measure
Next month failures fall.
The following month, failures fall from 34 to 15, and pending reboots fall most. Aisha reports the change with the caveat that one month may vary. The disk figure falls more slowly, because two disk upgrades are still waiting for delivery. Aisha keeps the monthly chart of failures by cause, so Gareth can see whether each fix holds after the first month. She notes that one month is too short to call the fall permanent, so the grouping continues each month.
Pending reboots
Updates often need a restart to complete. Staff postpone restarts because they are busy. A forced restart after seven days' notice, introduced in chapter 24, addresses this. Aisha also suggests a friendly reminder on day five.
For full disks, a scheduled clean-up removes temporary files; two laptops need larger disks. For offline laptops, internet-based management removes the dependence on the VPN.
Unknown failures
The two unknown failures are investigated individually; one is a corrupted update cache, fixed by resetting it.
Reading the breakdown
The 34 failures split into reboot 14, disk 9, offline 6, driver 3 and unknown 2. Most have simple, routine causes. Aisha explains that this is common: update failures are mostly operational, not mysterious, and grouping them by cause turns a frustrating list into a short set of fixes. She records the breakdown monthly so trends can be seen.
Project log entry
Aisha draws a simple chart of failures by cause each month. After the fixes, the reboot bar shrinks most sharply, the disk bar more slowly as two disk upgrades wait for delivery. The chart makes the effect of each fix visible to Gareth without any statistics beyond counting.
Worked example
Gareth was planning to buy a new patch tool. The analysis shows most failures come from reboots and disks, which policy changes fix without new tools.
Synthetic case. Synthetic.
Errors and limits
Tools rarely fix process causes.
Exercises
- EX082 — Top cause. What was the top failure cause? Rubric: 3 = pending reboot 41%; 2 = partial; 1 = vague; 0 = none.
- EX083 — Second. What was second? Rubric: 3 = disk full 26%; 2 = partial; 1 = vague; 0 = none.
- EX084 — Tool decision. Why not buy a new tool? Rubric: 3 = causes are process; 2 = partial; 1 = vague; 0 = none.
Synthesis
Analyse failures before buying solutions.
Continue studying — chapter 28 of 40
← Chapter 27 (draft) · Course chapter index · Chapter 29 (draft) →