## Chapter 1 — Computing systems ### Learning outcomes and connection After this chapter you should be able to: - describe a host in terms of the four resources an operator actually runs out of — compute, memory, storage and I/O — and the one resource people forget, **time**; - forecast when a resource will be exhausted from a short series of measurements, and say how far to trust the forecast; - explain why a host that serves an essential function needs **headroom**, not just "enough"; - recognize the common ways a host fails quietly, and write the first checks that would notice them. The entry course taught you how a computer works. This chapter asks a narrower question that runs through the whole book: *what must stay true about this computer for the service on top of it to keep working, and how would you know if it stopped being true?* LANTERN's relay host, RLY-01, is the first machine you will meet; by Chapter 29 it will be one of three. ### 1.1 A host as a set of finite resources Every service you run sits on a host with finite resources. **Compute** (CPU time) limits how much work can be done per second. **Memory** limits how much can be held at once; when it runs out, the operating system either swaps to disk (and everything slows) or terminates processes. **Storage** limits what can be kept; when a disk fills, writes fail, and many programs fail badly — a database may stop accepting writes, a logging daemon may silently drop messages, a relay may crash. **I/O** (disk and network throughput) limits how fast data can move in and out. The resource people forget is **time**: the host's clock. A wrong clock does not stop a service, but it makes its records disagree with everyone else's, breaks certificate validation and time-window checks, and makes investigations misleading. Chapters 17 and 20 return to time in detail. For a critical service, the useful question about each resource is not "how much is used?" but "how long until this resource stops the service, and how much warning will we get?" ### 1.2 Utilization, saturation and errors A practical way to inspect any resource is to ask three things about it: its **utilization** (how busy it is, as a fraction of capacity), its **saturation** (how much work is waiting because it is busy — a run queue, a disk queue, dropped packets) and its **errors** (failed operations). A CPU at 60 % utilization with no queue is fine; a disk at 40 % utilization with a long queue is not. Utilization alone hides the problem that users feel, which is waiting. Two habits follow. First, measure at a percentile, not an average: the 95th-percentile CPU over a day tells you about the busy moments that matter; the daily mean hides them. Second, measure over a long enough period to see the service's rhythm. Telemetry relays are busier when reports run and when stations batch-upload after a network outage; hospital systems peak at shift changes; transit systems peak at rush hour. ### 1.3 Headroom and why "enough" is not enough **Headroom** is the spare capacity between normal peak load and the point where the service degrades. An essential service needs headroom for three reasons: demand rises unexpectedly (a storm, a mass casualty event, an outage elsewhere that sends traffic to you); recovery itself consumes resources (stations that were offline upload their backlog when they return; a restore writes heavily to disk); and security tools consume resources (a scan, an integrity check or an antivirus update can double CPU use for minutes). A host sized to "just enough" for a normal day is sized to fail on the day it matters most. ### 1.4 Forecasting exhaustion For steadily growing resources — disk filled by logs and data is the classic case — a straight-line fit through recent daily measurements gives a usable forecast: growth per day, and days until an alarm threshold and until full. It is a forecast, not a promise. It assumes growth continues at the recent rate. It will be wrong if retention is changed, if a noisy process starts logging at debug level, or if an outage is followed by a burst of backlog. Use it to schedule action well before the threshold, and re-run it after any change. A useful complementary calculation is the **steady-state** size: if data are kept for *R* days and arrive at *g* GB per day, the retained volume levels off at about *g × R*. If that number exceeds the disk, no amount of "cleaning up" will save the host; either retention, volume or capacity must change. ### 1.5 Quiet failure modes Hosts in long-running services tend to fail in a small number of quiet ways: a disk or partition fills; a log rotation job stops and one file grows without limit; a certificate expires; the clock drifts after its time source becomes unreachable; a process leaks memory and is restarted by the system manager every few days, losing in-flight data; a scheduled job (backup, report) fails and nobody reads its email. Each of these can be noticed with a check that costs almost nothing — if somebody writes the check and somebody reads its result. Chapter 19 turns such checks into monitoring. ### 1.6 Worked example: RLY-01's disk (synthetic) E. Strand asks you whether RLY-01, which buffers station telemetry and keeps its own logs for 30 days, needs a bigger disk before winter. You collect 14 days of disk use (200 GB disk) and the daily 95th-percentile CPU. The disk grows by about 1.9 GB per day and is at 120.9 GB. The operations alarm fires at 85 % (170 GB). At this rate the alarm fires in about 26 days and the disk is full in about 41 days. At 30-day retention, the steady-state log volume is about 57 GB — but the disk is already well past that, which tells you the growth is not only logs within retention. Something is keeping data longer than 30 days. That is a configuration question, not a capacity question, and it should be answered before buying a disk. The CPU series shows a weekly pattern: days 6–7 of each week run at about 62 % against about 42 % on other days. You ask Water operations and learn that weekly compliance reports run then. This is normal load, but it is the load you must size for. ### 1.7 Practical activity: capacity headroom Run Lab 1. It fits a straight line to the disk series, reports days until the alarm and until full, the steady-state log volume at 30-day retention, and compares CPU on report days with other days. **Lab 1** (save as `lab01.py`, run with `python3 lab01.py`): ```python # Lab 1 - capacity headroom for the relay host (offline, standard library only) # Synthetic: Fallowmere relay host RLY-01. Daily samples over 14 days. disk_total_gb = 200.0 disk_used_gb = [96.0, 97.9, 99.7, 101.8, 103.6, 105.5, 107.6, 109.4, 111.3, 113.2, 115.1, 117.0, 118.8, 120.9] cpu_p95 = [41, 43, 40, 44, 42, 61, 63, 42, 41, 44, 43, 40, 62, 64] # percent, 95th percentile per day alarm_at = 0.85 # operations alarm when disk reaches 85 % full n = len(disk_used_gb) xs = list(range(n)) mx, my = sum(xs) / n, sum(disk_used_gb) / n slope = sum((x - mx) * (y - my) for x, y in zip(xs, disk_used_gb)) / sum((x - mx) ** 2 for x in xs) print(f"growth: {slope:.2f} GB/day") for limit, label in ((alarm_at * disk_total_gb, "85% alarm"), (disk_total_gb, "full")): days = (limit - disk_used_gb[-1]) / slope print(f"days until {label}: {days:.1f}") # retention: logs kept 30 days; growth comes from logs (synthetic assumption) print(f"steady-state log volume at 30-day retention: {slope*30:.1f} GB") weekly = [cpu_p95[i] for i in range(n) if i % 7 in (5, 6)] weekday = [cpu_p95[i] for i in range(n) if i % 7 not in (5, 6)] print(f"cpu p95 mean weekday {sum(weekday)/len(weekday):.1f}% vs days 6-7 of each week {sum(weekly)/len(weekly):.1f}%") ``` Expected output (Python 3.13.5): ```text growth: 1.91 GB/day days until 85% alarm: 25.7 days until full: 41.4 steady-state log volume at 30-day retention: 57.3 GB cpu p95 mean weekday 42.0% vs days 6-7 of each week 62.5% ``` Record the four numbers in your working note under **observed**, then write one **inferred** sentence and one **open** item. A good open item names a question and an owner: "Why does RLY-01 retain more than 30 days of data? — E. Strand, by Friday." ### 1.8 Common errors and limits - Reporting average CPU and concluding there is headroom, when the peak — the moment that matters — is close to saturation. - Treating a linear forecast as a guarantee; it assumes the future looks like the last two weeks. - Solving a retention or logging fault by buying capacity, which hides the fault until the new disk fills. - Forgetting that recovery and security activity consume resources. - Setting an alarm at a threshold that leaves less warning time than it takes to procure, approve and fit more capacity. - Ignoring the clock because "it's only time". Fourteen samples are a small basis for any forecast. The method's value is that it turns an impression ("the disk seems to be filling") into a number with an explicit assumption that someone can check. ### Exercises — chapter 1 **EX001 — Lead time versus warning time.** *Objective:* set an alarm threshold from operational lead time. *Inputs:* Lab 1 output; procurement and approved fitting of a new disk takes 21 working days (about 30 calendar days). *Constraints:* keep the same growth rate. *Steps:* (1) compute the fill level at which you must raise the request so that the disk is fitted before it is full with 7 days to spare; (2) express it as a percentage of the disk; (3) say whether the 85 % alarm is adequate. *Expected evidence:* calculation and two-sentence recommendation. *Rubric:* 3 = correct threshold (37 days × 1.91 GB ≈ 71 GB of free space needed, i.e. raise at about 129 GB or 65 %, which RLY-01 reaches in about four days) and a clear judgement that 85 % is too late for this lead time; 2 = method right, arithmetic slip; 1 = threshold chosen without lead time; 0 = none. **EX002 — Explain the growth.** *Objective:* separate a capacity problem from a configuration problem. *Inputs:* the worked example. *Constraints:* do not touch any real system; reason from the numbers. *Steps:* (1) compute how many days of data at 1.91 GB/day the current 120.9 GB represents; (2) give two plausible configuration causes; (3) describe the check that would distinguish them. *Expected evidence:* short note with observed / inferred / open labels. *Rubric:* 3 = about 63 days computed, two specific causes (for example rotation not deleting, buffer for an offline upstream not being purged, a second log stream outside the retention policy) and a check for each; 2 = causes generic; 1 = recommends a larger disk only; 0 = none. **EX003 — Quiet failure checklist.** *Objective:* turn quiet failure modes into checks. *Inputs:* Chapter 1 ("Quiet failure modes"). *Constraints:* five checks; each must name what is measured, the threshold and who reads the result. *Steps:* write a table with columns: failure mode, measurement, threshold, reader, action. *Expected evidence:* table. *Rubric:* 3 = five distinct failure modes, measurable thresholds with warning time, a named reader and an action for each; 2 = one column weak; 1 = generic "monitor the server"; 0 = none. ### Chapter synthesis A host is a set of finite resources plus a clock. For an essential service the question about each resource is how long until it stops the service and how much warning you will get. Measure utilization, saturation and errors at percentiles over the service's natural rhythm; keep headroom for bad days, recovery and security work; forecast exhaustion with explicit assumptions; and write cheap checks for the quiet failures that actually take services down. When numbers disagree with expectations, as RLY-01's disk did, look for a configuration cause before buying capacity. ### Continue studying — chapter 01 of 30 [← Course home](../../README.md) · [Course chapter index](../SUBJECTS.md) · [Chapter 02 →](#ch-02) Continue to the next authored lesson. ## Worked answers **EX001 — Lead time versus warning time.** Free space needed when the request is raised = (30 days lead + 7 days spare) × 1.91 GB/day ≈ 70.7 GB. Raise the request when used space reaches about 200 − 70.7 ≈ 129 GB, i.e. about 65 % full. RLY-01 is at 120.9 GB and will reach 129 GB in about (129.3 − 120.9) / 1.91 ≈ 4.4 days. The 85 % alarm fires with only about 15.7 days of growth left before full — half the procurement lead time — so it is too late as the *trigger for buying*; keep it as an operational alarm and add a 65 % "raise capacity request" threshold. Full credit also notes that fixing retention (EX002) may make the purchase unnecessary. **EX002 — Explain the growth.** 120.9 GB ÷ 1.91 GB/day ≈ 63 days of data, more than twice the 30-day retention. *Observed:* disk use and growth rate. *Inferred:* data older than 30 days are being kept. Plausible causes: (a) log rotation runs but old files are not deleted (rotation count set too high, or compression without deletion); (b) the relay's upstream buffer kept messages during an old historian outage and never purged them; (c) a second log stream (debug or audit) is outside the rotation policy. Checks (read-only): list the oldest files by directory and modification date; compare the rotation configuration with policy; check the buffer directory size and oldest entry. *Open:* cause confirmed — E. Strand, by Friday. **EX003 — Quiet failure checklist.** Example rows: (1) disk filling — % used and days-to-full forecast — warn at 65 %/alarm at 85 % — platform on-call — raise capacity or fix retention; (2) log rotation stopped — age of the oldest active log file — > 2 days — platform on-call — fix rotation; (3) certificate expiry — days to expiry of relay and historian certificates — < 30 days — platform team — renew under standard change; (4) clock drift — offset from internal NTP — > 1 s warn, > 2 s alarm — on-call — check time source; (5) failed scheduled job — backup job result and restore-test age — any failure, or restore test > 90 days — service owner — rerun and investigate. Full credit requires measurable thresholds with warning time and a named reader.