Chapter 17 — Control and feedback basics
Lesson map
Lesson 17 published. Map, example, EX049.
flowchart TD
N0["Learning outcomes"]
N1["17.1 Feedback loops"]
N2["17.2 Quality control in labs"]
N3["17.3 Metrics"]
N4["17.4 Delay and oscillation"]
N5["17.5 Goodhart's problem"]
N6["17.6 Corrective action"]
N7["17.7 Visualising control"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX049"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["17.1 Feedback loops"]
N2["17.2 Quality control in labs"]
N3["17.3 Metrics"]
N4["17.4 Delay and oscillation"]
N5["17.5 Goodhart's problem"]
N6["17.6 Corrective action"]
N7["17.7 Visualising control"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX049"]
[Learning outcomes]
|
v
[17.1 Feedback loops]
|
v
[17.2 Quality control in labs]
|
v
[17.3 Metrics]
|
v
[17.4 Delay and oscillation]
|
v
[17.5 Goodhart's problem]
|
v
[17.6 Corrective action]
|
v
[17.7 Visualising control]
|
v
[Worked example] --> [Exercise EX049]
Learning outcomes
You will (1) explain feedback loops; (2) apply them to lab quality control; (3) design metrics that drive improvement; (4) avoid metrics that distort behaviour.
17.1 Feedback loops
A controller measures output, compares it with a target and adjusts input. A thermostat is the classic example. Forensic quality works the same way: measure, compare with standard, correct.
The three parts need definitions. The measurement might be the share of reports that come back from peer review with substantive corrections. The target might be under five percent. The adjustment might be targeted training or a revised procedure. Without a defined measurement, the loop runs on impressions, and impressions tend to favour whichever problem was most recent or most embarrassing.
17.2 Quality control in labs
Accreditation schemes for testing laboratories expect monitoring of results — for example proficiency tests, where examiners analyse prepared samples whose answers are known. Results feed back into training and procedure changes.
A proficiency test might give each examiner a synthetic image with planted artefacts and ask specific questions, such as when a file was created and whether a USB device was connected. Answers are compared with the known truth. A wrong answer leads to a review of how the examiner reached it, which often shows a procedure gap rather than individual carelessness.
17.3 Metrics
Turnaround time, rework rate, verification failures and peer-review findings are measurable. Each needs a definition precise enough that two people compute the same value.
Turnaround time needs a start and an end: does it start at receipt or at assignment, and end at the draft report or at the signed one? Two teams using different definitions can report very different numbers for the same work. Priya writes each metric as a short specification, including the data source and how exceptions such as paused cases are handled, and keeps it under version control.
17.4 Delay and oscillation
Feedback with long delay can overcorrect. If quarterly turnaround figures trigger sudden staffing moves, the backlog may swing back and forth. Smaller, more frequent adjustments help.
If managers move two examiners to the backlog after a bad quarter, the backlog falls, the examiners move back, and three months later it rises again. The system swings because the correction arrives long after the measurement. Monthly figures with smaller adjustments, or a standing allocation sized to the average arrival rate, dampen the swing. Priya demonstrates this with a short simulation.
17.5 Goodhart's problem
When a measure becomes a target, people optimise the measure. Rewarding fast turnaround alone can push examiners to skip checks. Pair speed metrics with quality metrics.
A turnaround target can also encourage splitting cases so each part closes quickly, or delaying the start date until an examiner is ready to finish fast. Pairing the speed measure with rework rate and peer-review corrections makes shortcuts show up somewhere. Priya also suggests reviewing a small random sample of closed cases each month, which is hard to game.
17.6 Corrective action
When a defect is found, the loop includes root-cause analysis and a check that the fix worked.
Root-cause analysis asks why a defect happened until it reaches something the lab can change. A misattributed timestamp might come from an examiner forgetting a time-zone offset, but the underlying cause may be a tool that displays local time by default. The fix changes the default, and the check confirms that later cases show UTC. Without that check, corrective action is only intention.
17.7 Visualising control
Control charts show a metric over time with expected limits, making unusual shifts visible.
Control limits are usually set from the metric's own past variation, often three standard deviations from the mean. A single point beyond a limit, or several points in a row on one side of the mean, suggests a real change rather than noise. Priya plots monthly rework rates for two synthetic years and marks the month a new tool version arrived, where the chart shows a visible shift.
Feedback from courts
Court outcomes and questions raised in cross-examination are another feedback source. When a defence expert challenges a timestamp interpretation, the lab reviews whether its procedure explained the artefact clearly enough and, if not, updates the procedure and trains staff, then checks later reports for the improvement over the next quarter.
Worked example
Northfield's synthetic data show turnaround improved 20% after a target was set, but peer-review findings doubled. Priya recommends adding a combined indicator and monthly review, rather than abandoning the target.
Synthetic case. The figures are invented.
Errors and limits
Metrics capture only what they measure; regular case file reviews catch what metrics miss.
Exercises
- EX049 — Proficiency test. How does a proficiency test act as feedback? Rubric: 3 = known answers measure accuracy, results drive training/procedure change; 2 = partial; 1 = exam; 0 = none.
- EX050 — Metric distortion. What happened when only turnaround was targeted? Rubric: 3 = faster but review findings doubled; 2 = partial; 1 = improved; 0 = none.
- EX051 — Delay effect. Why can long feedback delays cause oscillation? Rubric: 3 = overcorrection after delayed signal; 2 = partial; 1 = vague; 0 = none.
Synthesis
Quality is a feedback loop: measure well, pair speed with quality, adjust in small steps.
Continue studying — chapter 17 of 50
← Chapter 16 (draft) · Course chapter index · Chapter 18 (draft) →