Chapter 17 — Control and feedback basics
Lesson map
Lesson 17 published. Map, example, EX049.
flowchart TD
N0["Learning outcomes"]
N1["17.1 Feedback and stability"]
N2["17.2 Feedback loops in AI systems"]
N3["17.3 Safe operating envelopes"]
N4["17.4 Monitoring loops"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX049"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["17.1 Feedback and stability"]
N2["17.2 Feedback loops in AI systems"]
N3["17.3 Safe operating envelopes"]
N4["17.4 Monitoring loops"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX049"]
[Learning outcomes]
|
v
[17.1 Feedback and stability]
|
v
[17.2 Feedback loops in AI systems]
|
v
[17.3 Safe operating envelopes]
|
v
[17.4 Monitoring loops]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX049]
Learning outcomes
You will (1) describe open- and closed-loop control, feedback, gain and stability; (2) recognise feedback loops in AI systems (retraining on outputs, recommender loops, agents acting on their own results); (3) explain how adversaries exploit feedback; and (4) design safe operating envelopes and interlocks for AI-driven control.
17.1 Feedback and stability
A closed-loop controller measures output, compares it with a target and adjusts input. Gain determines how strongly it reacts; too much gain causes oscillation or instability, too little causes slow response. Delays in the loop make instability more likely.
A thermostat is a familiar loop: it measures room temperature and switches heating on or off. If it reacts to every small fluctuation, heating toggles rapidly; if it reacts too slowly, the room swings between hot and cold. Delay worsens both, because the controller acts on information that is already out of date. The same trade-offs appear wherever a system adjusts itself from its own results, including automated threshold tuning.
17.2 Feedback loops in AI systems
AI systems often form loops: a model's decisions change the data it is later retrained on; an anomaly model whose alerts trigger maintenance never sees what would have happened without maintenance; an agent observes the results of its own tool calls and plans further actions. Loops can amplify small errors — and amplify an attacker's influence. If attackers can inject data that is fed back into training (for example user feedback "thumbs up" on manipulated answers), they can steer the model over time; this is a poisoning path (OWASP LLM04).
17.3 Safe operating envelopes
Control engineers bound actuator commands, rate-limit changes, and add interlocks that override the controller when measurements leave a safe region. Apply the same to AI: bound the actions an agent may take per hour, require human approval outside an envelope, and add independent checks (a simple rule-based monitor) that can veto the model.
The rule-based monitor is deliberately simple, so it fails in different ways from the model. A rule such as 'no more than five work orders per hour from one session' catches runaway behaviour regardless of why the model produced it. Envelopes are written from operating data: the normal range of actions observed, with a margin, and an explicit owner who approves changes.
17.4 Monitoring loops
Monitor input distribution, output distribution and outcome rates. Sudden changes may indicate drift or attack; slow drift may indicate a feedback loop. Keep a reference dataset that is not influenced by the model to measure true performance.
A reference set is labelled independently and kept away from training and tuning. Measuring the model on it at intervals shows true performance, even when production feedback is biased by the model's own decisions. Monitoring outcome rates also needs care: if maintenance follows every alert, the system never sees missed failures directly, so engineers sample unalerted cases for inspection to estimate them.
17.5 Worked example (synthetic NRS case)
Pulse's thresholds were automatically retuned weekly from technician feedback on alerts. A contractor team, under schedule pressure, marked many true alerts as "false alarm"; over six weeks the threshold rose and true-positive rate fell from 92% to 78% on the reference set. Sam replaced automatic retuning with a bounded update (at most 2% threshold change per week), a reference-set check before each change, and review of feedback from any team whose feedback diverged sharply from others.
Without the reference set, the decline would have looked like improvement, since fewer alerts were being raised.
17.6 Errors and limits
- Retraining on model-influenced data without a clean reference.
- Unbounded automatic updates.
- Agents with no action budget.
Retraining on data shaped by the model's own decisions can lock in its errors, because cases it missed never come back as examples. Unbounded updates let a few biased inputs move the system far before anyone notices. Agents without action budgets can repeat a harmful action many times in the minutes before a human reviews the log. Each mistake is a missing bound on a feedback loop, and each fix is a limit that does not depend on the model being right.
Exercises
- EX049 — Loops. Identify two feedback loops in Wrench. Rubric: 3 = two plausible loops (feedback ratings → fine-tuning; agent observing tool results) with risks; 2 = one; 1 = vague; 0 = none.
- EX050 — Envelope. Propose three envelope limits for an agent that creates work orders. Rubric: 3 = per-period cap, approval outside scope, independent veto/monitor; 2 = two; 1 = one; 0 = none.
- EX051 — Pulse fix. Explain why the reference set is essential in Sam's fix. Rubric: 3 = it is not influenced by feedback and reveals true performance decline; 2 = partial; 1 = vague; 0 = wrong.
Synthesis
Feedback makes systems adaptive and also exploitable. Bounded updates, clean references, action envelopes and independent interlocks keep AI loops stable under attack.
Continue studying — chapter 17 of 50
← Chapter 16 (draft) · Course chapter index · Chapter 18 (draft) →