Chapter 17 — Control and feedback basics
Lesson map
Lesson 17 published. Map, example, EX049.
flowchart TD
N0["Learning outcomes"]
N1["17.1 Feedback loops"]
N2["17.2 Autoscaling rule"]
N3["Lab — Lab — autoscaling loop"]
N4["17.3 Lab result"]
N5["17.4 Stability"]
N6["17.5 Security and feedback"]
N7["Tuning"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX049"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["17.1 Feedback loops"]
N2["17.2 Autoscaling rule"]
N3["Lab — Lab — autoscaling loop"]
N4["17.3 Lab result"]
N5["17.4 Stability"]
N6["17.5 Security and feedback"]
N7["Tuning"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX049"]
[Learning outcomes]
|
v
[17.1 Feedback loops]
|
v
[17.2 Autoscaling rule]
|
v
[Lab — Lab — autoscaling loop]
|
v
[17.3 Lab result]
|
v
[17.4 Stability]
|
v
[17.5 Security and feedback]
|
v
[Tuning]
|
v
[Worked example] --> [Exercise EX049]
Learning outcomes
You will (1) describe feedback loops in cloud systems; (2) analyse an autoscaling rule; (3) reason about stability, limits and oscillation; (4) relate reconciliation loops to security.
17.1 Feedback loops
A controller measures an output, compares it with a target and adjusts an input. Kubernetes is built from such loops: controllers continually reconcile the actual state toward the declared desired state, as Burns et al. (2016) describe.
Each Kubernetes controller watches one kind of object. The deployment controller sees that three replicas are declared but two are running and starts another. The same idea makes the system self-healing: a crashed pod, a lost node or a deleted object is noticed and corrected without a person intervening. Understanding this helps engineers predict how the cluster will respond to failures and to manual changes.
17.2 Autoscaling rule
The horizontal pod autoscaler sets desired replicas = ceil(current replicas × current metric ÷ target metric), within minimum and maximum bounds.
The formula scales replicas in proportion to how far the metric is from its target. If CPU runs at twice the target, the autoscaler roughly doubles the replicas. Rounding up with the ceiling function means it errs towards more capacity. The bounds stop the formula from shrinking a service to nothing at night or growing it without limit during a surge.
Lab — Lab — autoscaling loop
"""Replica feedback rule desired = ceil(current * metric / target), with min/max bounds (stdlib only)."""
import math
target, lo, hi = 60, 2, 10 # target CPU %, min and max replicas
replicas = 2
for load in [50, 90, 150, 200, 120, 40]: # synthetic CPU % per replica at current size
desired = max(lo, min(hi, math.ceil(replicas * load / target)))
print(f"util {load:3}% at {replicas:2} replicas -> desired {desired}")
replicas = desired
Run with python3 lab_autoscale.py. Verified output:
util 50% at 2 replicas -> desired 2
util 90% at 2 replicas -> desired 3
util 150% at 3 replicas -> desired 8
util 200% at 8 replicas -> desired 10
util 120% at 10 replicas -> desired 10
util 40% at 10 replicas -> desired 7
17.3 Lab result
With a 60% CPU target and bounds of 2 to 10: at 50% on 2 replicas it stays at 2; at 90% it scales to 3; at 150% on 3 to 8; at 200% on 8 it would want 27 but is capped at 10; at 120% on 10 it stays at the cap of 10; at 40% on 10 it scales down to 7. The cap protects cost and downstream systems but means sustained overload is possible.
Working each case by hand shows how the formula behaves. At 150% on three replicas, 3 × 150 ÷ 60 is 7.5, rounded up to 8. At 200% on eight, the formula asks for 27 but the bound holds it at 10. At 40% on ten, 10 × 40 ÷ 60 is about 6.7, rounded up to 7. Elena records the cap as a deliberate risk: if real demand exceeds ten replicas' capacity, readings will queue.
17.4 Stability
Loops that react too fast oscillate; real autoscalers add stabilisation windows so scale-down waits. Measurement delay makes oscillation more likely.
Without a stabilisation window, a brief dip in load could remove pods that are needed again a minute later, causing repeated scale-up and scale-down. Each new pod takes time to start, so oscillation wastes resources and can briefly reduce capacity. Waiting several minutes before scaling down smooths the response. Metrics collected every thirty seconds add their own delay, which the window must allow for.
17.5 Security and feedback
Reconciliation also restores security settings: if someone edits a live object by hand, a GitOps controller changes it back to the declared state. Security alerts and responses form a loop too; slow detection is like measurement delay.
Reconciliation turns the declared configuration into an enforcement mechanism: a hand edit that weakens a setting is reverted within minutes. It also means emergency fixes must go through version control, or the controller will undo them. On the detection side, long delays between an event and an alert act like slow measurement in a control loop, making the response arrive after the damage.
17.6 Limits as safety
Maximum replicas also limit the cost of a flood of fake traffic, a simple availability safeguard.
Without a maximum, a flood of fake requests could drive the autoscaler to create hundreds of pods, running up a large bill and possibly overwhelming the database. The cap turns an unbounded cost into a bounded one. The trade-off is that genuine extreme demand is also capped, so the maximum is set from measured peaks with a margin and reviewed as meter numbers grow.
Tuning
Stabilisation windows are tuned using load-test data rather than guesses.
Worked example
In a load test, ingest hits the 10-replica cap for 40 minutes and the queue grows. Elena recommends raising the cap to 14 after checking the database can handle the extra connections, and adding an alert when the cap is reached for more than 10 minutes.
Synthetic case. Loads and replicas are synthetic.
Errors and limits
A capped autoscaler hides overload unless the cap itself is monitored.
Exercises
- EX049 — Formula. Write the autoscaling formula. Rubric: 3 = ceil(current × metric / target) within bounds; 2 = partial; 1 = vague; 0 = none.
- EX050 — Capped step. What happened at 200% on 8 replicas? Rubric: 3 = wanted 27, capped at 10; 2 = partial; 1 = vague; 0 = none.
- EX051 — Reconciliation security. How does reconciliation help security? Rubric: 3 = reverts manual edits to declared state; 2 = partial; 1 = vague; 0 = none.
Synthesis
Feedback loops scale systems and restore declared security state.
Continue studying — chapter 17 of 50
← Chapter 16 (draft) · Course chapter index · Chapter 18 (draft) →