Chapter 1 — Computing systems
Lesson map
Lesson 01 published. Map, example, EX001.
flowchart TD
N0["Learning outcomes and connection"]
N1["1.1 A host as a set of finite resources"]
N2["1.2 Utilization, saturation and errors"]
N3["1.3 Headroom and why enough is not enough"]
N4["1.4 Forecasting exhaustion"]
N5["1.5 Quiet failure modes"]
N6["1.7 Practical activity: capacity headroom"]
N7["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX001"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["1.1 A host as a set of finite resources"]
N2["1.2 Utilization, saturation and errors"]
N3["1.3 Headroom and why enough is not enough"]
N4["1.4 Forecasting exhaustion"]
N5["1.5 Quiet failure modes"]
N6["1.7 Practical activity: capacity headroom"]
N7["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX001"]
[Learning outcomes and connection]
|
v
[1.1 A host as a set of finite resources]
|
v
[1.2 Utilization, saturation and errors]
|
v
[1.3 Headroom and why enough is not enough]
|
v
[1.4 Forecasting exhaustion]
|
v
[1.5 Quiet failure modes]
|
v
[1.7 Practical activity: capacity headroom]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX001]
Learning outcomes and connection
After this chapter you should be able to:
- describe a host in terms of the four resources an operator actually runs out of — compute, memory, storage and I/O — and the one resource people forget, time;
- forecast when a resource will be exhausted from a short series of measurements, and say how far to trust the forecast;
- explain why a host that serves an essential function needs headroom, not just "enough";
- recognize the common ways a host fails quietly, and write the first checks that would notice them.
The entry course taught you how a computer works. This chapter asks a narrower question that runs through the whole book: what must stay true about this computer for the service on top of it to keep working, and how would you know if it stopped being true? LANTERN's relay host, RLY-01, is the first machine you will meet; by Chapter 29 it will be one of three.
1.1 A host as a set of finite resources
Every service you run sits on a host with finite resources. Compute (CPU time) limits how much work can be done per second. Memory limits how much can be held at once; when it runs out, the operating system either swaps to disk (and everything slows) or terminates processes. Storage limits what can be kept; when a disk fills, writes fail, and many programs fail badly — a database may stop accepting writes, a logging daemon may silently drop messages, a relay may crash. I/O (disk and network throughput) limits how fast data can move in and out.
The resource people forget is time: the host's clock. A wrong clock does not stop a service, but it makes its records disagree with everyone else's, breaks certificate validation and time-window checks, and makes investigations misleading. Chapters 17 and 20 return to time in detail.
For a critical service, the useful question about each resource is not "how much is used?" but "how long until this resource stops the service, and how much warning will we get?"
1.2 Utilization, saturation and errors
A practical way to inspect any resource is to ask three things about it: its utilization (how busy it is, as a fraction of capacity), its saturation (how much work is waiting because it is busy — a run queue, a disk queue, dropped packets) and its errors (failed operations). A CPU at 60 % utilization with no queue is fine; a disk at 40 % utilization with a long queue is not. Utilization alone hides the problem that users feel, which is waiting.
Two habits follow. First, measure at a percentile, not an average: the 95th-percentile CPU over a day tells you about the busy moments that matter; the daily mean hides them. Second, measure over a long enough period to see the service's rhythm. Telemetry relays are busier when reports run and when stations batch-upload after a network outage; hospital systems peak at shift changes; transit systems peak at rush hour.
1.3 Headroom and why "enough" is not enough
Headroom is the spare capacity between normal peak load and the point where the service degrades. An essential service needs headroom for three reasons: demand rises unexpectedly (a storm, a mass casualty event, an outage elsewhere that sends traffic to you); recovery itself consumes resources (stations that were offline upload their backlog when they return; a restore writes heavily to disk); and security tools consume resources (a scan, an integrity check or an antivirus update can double CPU use for minutes). A host sized to "just enough" for a normal day is sized to fail on the day it matters most.
1.4 Forecasting exhaustion
For steadily growing resources — disk filled by logs and data is the classic case — a straight-line fit through recent daily measurements gives a usable forecast: growth per day, and days until an alarm threshold and until full. It is a forecast, not a promise. It assumes growth continues at the recent rate. It will be wrong if retention is changed, if a noisy process starts logging at debug level, or if an outage is followed by a burst of backlog. Use it to schedule action well before the threshold, and re-run it after any change.
A useful complementary calculation is the steady-state size: if data are kept for R days and arrive at g GB per day, the retained volume levels off at about g × R. If that number exceeds the disk, no amount of "cleaning up" will save the host; either retention, volume or capacity must change.
1.5 Quiet failure modes
Hosts in long-running services tend to fail in a small number of quiet ways: a disk or partition fills; a log rotation job stops and one file grows without limit; a certificate expires; the clock drifts after its time source becomes unreachable; a process leaks memory and is restarted by the system manager every few days, losing in-flight data; a scheduled job (backup, report) fails and nobody reads its email. Each of these can be noticed with a check that costs almost nothing — if somebody writes the check and somebody reads its result. Chapter 19 turns such checks into monitoring.
1.6 Worked example: RLY-01's disk (synthetic)
E. Strand asks you whether RLY-01, which buffers station telemetry and keeps its own logs for 30 days, needs a bigger disk before winter. You collect 14 days of disk use (200 GB disk) and the daily 95th-percentile CPU. The disk grows by about 1.9 GB per day and is at 120.9 GB. The operations alarm fires at 85 % (170 GB). At this rate the alarm fires in about 26 days and the disk is full in about 41 days. At 30-day retention, the steady-state log volume is about 57 GB — but the disk is already well past that, which tells you the growth is not only logs within retention. Something is keeping data longer than 30 days. That is a configuration question, not a capacity question, and it should be answered before buying a disk.
The CPU series shows a weekly pattern: days 6–7 of each week run at about 62 % against about 42 % on other days. You ask Water operations and learn that weekly compliance reports run then. This is normal load, but it is the load you must size for.
1.7 Practical activity: capacity headroom
Run Lab 1. It fits a straight line to the disk series, reports days until the alarm and until full, the steady-state log volume at 30-day retention, and compares CPU on report days with other days.
Lab 1 (save as lab01.py, run with python3 lab01.py):
# Lab 1 - capacity headroom for the relay host (offline, standard library only)
# Synthetic: Fallowmere relay host RLY-01. Daily samples over 14 days.
disk_total_gb = 200.0
disk_used_gb = [96.0, 97.9, 99.7, 101.8, 103.6, 105.5, 107.6, 109.4, 111.3, 113.2, 115.1, 117.0, 118.8, 120.9]
cpu_p95 = [41, 43, 40, 44, 42, 61, 63, 42, 41, 44, 43, 40, 62, 64] # percent, 95th percentile per day
alarm_at = 0.85 # operations alarm when disk reaches 85 % full
n = len(disk_used_gb)
xs = list(range(n))
mx, my = sum(xs) / n, sum(disk_used_gb) / n
slope = sum((x - mx) * (y - my) for x, y in zip(xs, disk_used_gb)) / sum((x - mx) ** 2 for x in xs)
print(f"growth: {slope:.2f} GB/day")
for limit, label in ((alarm_at * disk_total_gb, "85% alarm"), (disk_total_gb, "full")):
days = (limit - disk_used_gb[-1]) / slope
print(f"days until {label}: {days:.1f}")
# retention: logs kept 30 days; growth comes from logs (synthetic assumption)
print(f"steady-state log volume at 30-day retention: {slope*30:.1f} GB")
weekly = [cpu_p95[i] for i in range(n) if i % 7 in (5, 6)]
weekday = [cpu_p95[i] for i in range(n) if i % 7 not in (5, 6)]
print(f"cpu p95 mean weekday {sum(weekday)/len(weekday):.1f}% vs days 6-7 of each week {sum(weekly)/len(weekly):.1f}%")
Expected output (Python 3.13.5):
growth: 1.91 GB/day
days until 85% alarm: 25.7
days until full: 41.4
steady-state log volume at 30-day retention: 57.3 GB
cpu p95 mean weekday 42.0% vs days 6-7 of each week 62.5%
Record the four numbers in your working note under observed, then write one inferred sentence and one open item. A good open item names a question and an owner: "Why does RLY-01 retain more than 30 days of data? — E. Strand, by Friday."
1.8 Common errors and limits
- Reporting average CPU and concluding there is headroom, when the peak — the moment that matters — is close to saturation.
- Treating a linear forecast as a guarantee; it assumes the future looks like the last two weeks.
- Solving a retention or logging fault by buying capacity, which hides the fault until the new disk fills.
- Forgetting that recovery and security activity consume resources.
- Setting an alarm at a threshold that leaves less warning time than it takes to procure, approve and fit more capacity.
- Ignoring the clock because "it's only time".
Fourteen samples are a small basis for any forecast. The method's value is that it turns an impression ("the disk seems to be filling") into a number with an explicit assumption that someone can check.
Exercises — chapter 1
EX001 — Lead time versus warning time. Objective: set an alarm threshold from operational lead time. Inputs: Lab 1 output; procurement and approved fitting of a new disk takes 21 working days (about 30 calendar days). Constraints: keep the same growth rate. Steps: (1) compute the fill level at which you must raise the request so that the disk is fitted before it is full with 7 days to spare; (2) express it as a percentage of the disk; (3) say whether the 85 % alarm is adequate. Expected evidence: calculation and two-sentence recommendation. Rubric: 3 = correct threshold (37 days × 1.91 GB ≈ 71 GB of free space needed, i.e. raise at about 129 GB or 65 %, which RLY-01 reaches in about four days) and a clear judgement that 85 % is too late for this lead time; 2 = method right, arithmetic slip; 1 = threshold chosen without lead time; 0 = none.
EX002 — Explain the growth. Objective: separate a capacity problem from a configuration problem. Inputs: the worked example. Constraints: do not touch any real system; reason from the numbers. Steps: (1) compute how many days of data at 1.91 GB/day the current 120.9 GB represents; (2) give two plausible configuration causes; (3) describe the check that would distinguish them. Expected evidence: short note with observed / inferred / open labels. Rubric: 3 = about 63 days computed, two specific causes (for example rotation not deleting, buffer for an offline upstream not being purged, a second log stream outside the retention policy) and a check for each; 2 = causes generic; 1 = recommends a larger disk only; 0 = none.
EX003 — Quiet failure checklist. Objective: turn quiet failure modes into checks. Inputs: Chapter 1 ("Quiet failure modes"). Constraints: five checks; each must name what is measured, the threshold and who reads the result. Steps: write a table with columns: failure mode, measurement, threshold, reader, action. Expected evidence: table. Rubric: 3 = five distinct failure modes, measurable thresholds with warning time, a named reader and an action for each; 2 = one column weak; 1 = generic "monitor the server"; 0 = none.
Chapter synthesis
A host is a set of finite resources plus a clock. For an essential service the question about each resource is how long until it stops the service and how much warning you will get. Measure utilization, saturation and errors at percentiles over the service's natural rhythm; keep headroom for bad days, recovery and security work; forecast exhaustion with explicit assumptions; and write cheap checks for the quiet failures that actually take services down. When numbers disagree with expectations, as RLY-01's disk did, look for a configuration cause before buying capacity.
Continue studying — chapter 01 of 30
← Course home · Course chapter index · Chapter 02 →
Continue to the next authored lesson.