Chapter 46 — Engineering capstone: trial and iteration
Lesson map
Lesson 46 published. Map, example, EX091.
flowchart TD
N0["Learning outcomes and connection"]
N1["46.1 Trial protocol"]
N2["46.2 Scenarios for response systems"]
N3["46.3 Instrumentation"]
N4["46.4 Defects and prioritisation"]
N5["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX091"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["46.1 Trial protocol"]
N2["46.2 Scenarios for response systems"]
N3["46.3 Instrumentation"]
N4["46.4 Defects and prioritisation"]
N5["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX091"]
[Learning outcomes and connection]
|
v
[46.1 Trial protocol]
|
v
[46.2 Scenarios for response systems]
|
v
[46.3 Instrumentation]
|
v
[46.4 Defects and prioritisation]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX091]
Learning outcomes and connection
After this chapter you can (1) write a trial protocol for a capstone response system under controlled scenarios; (2) instrument the system to measure what the protocol requires; (3) classify and prioritise defects found in trials; and (4) iterate the design and rerun acceptance tests with change-impact analysis. It builds on Chapters 29, 40, 44 and 45 and prepares Chapters 47–50.
46.1 Trial protocol
A trial is a planned set of scenarios run against the built system with measurement. The protocol states: scenarios and their parameters (telemetry completeness, approval delay, volume); success criteria per requirement; instrumentation; number of runs and seeds; who plays which role (approvers can be students acting from scripts); stop conditions; and how results will be analysed. Pre-specify it — the discipline of Chapter 29 applies. The HVSS-style trial below fixes these in advance: six scenarios, five seeded runs each, synthetic ground truth.
46.2 Scenarios for response systems
Vary the conditions that stress the design: (S1) normal — complete telemetry, prompt approval; (S2) degraded telemetry — one source 20% missing, another delayed 8 minutes; (S3) approval delay — primary approver silent, deputy responds at 40 minutes; (S4) volume spike — 5× normal events for 15 minutes; (S5) dependency failure — evidence store unavailable for 10 minutes; (S6) adversarial input — conflicting record IDs and a forged tenant claim. Each scenario has synthetic ground truth so outcomes can be scored.
Several of these scenarios reuse earlier labs, now run at the scale of a whole system. S2's 8-minute delay is the delay that silenced the arrival-time rule in Lab 36, and S5 repeats the Lab 40 outage in shorter form. S6 combines the conflicting record IDs of Lab 30 with the forged claim of AT-07.
46.3 Instrumentation
Measure from the system's own records: event, ingest and queue timestamps; workflow transitions; action effects; manifest verification results; invariant checks; resource use. Instrumentation is part of the design: if a requirement cannot be measured from the system's records, either add instrumentation or rewrite the requirement.
In the HVSS-style trial, defect D3 was found because event-time lag p99 could be computed from ingest and queue timestamps. Without them, the 11-minute lag in S4 would have been invisible. Those timestamps are the same fields defined in the event schema of Chapter 22, now used as trial instruments.
46.4 Defects and prioritisation
Classify each defect by the requirement affected, severity (safety/security claim broken, objective missed, cosmetic), reproducibility and root cause (design, implementation, configuration, test). Prioritise safety and security first, then objectives, then usability. Each fix gets a regression test and a change-impact analysis: which requirements, interfaces and tests could the change affect? Rerun those tests, not only the one that failed.
In the HVSS-style trial, D2 is classified first even though it appeared in only one run: an action executed twice breaks a safety claim. Its fix, per-case serialisation with an idempotency key, triggered reruns of workflow and controller tests.
46.5 Worked example (synthetic): an HVSS-style trial
Six scenarios, five seeded runs each. Results: S1 all criteria met. S2: two correlations missed in 5 runs — rule evaluated on arrival time for one source (defect D1, design; fix: event-time evaluation with watermark; impact: R1 latency increases by watermark; reran AT-01 and S1). S3: escalation worked; one run showed the deputy's approval recorded but action executed twice (D2, implementation, concurrency between retry and approval; severity high; fix: per-case serialisation and idempotency key on actions; impact: workflow and controller tests rerun). S4: event-time lag p99 reached 11 minutes against 5 required (D3, objective missed; fix: batch size tuned and second consumer; reran S4: p99 4.2 minutes). S5: outbox worked, no loss. S6: forged tenant claim rejected; conflicting IDs stored with conflict event.
Iteration 2 reran all scenarios after fixes: all criteria met except S4 at 10× (out of scope, recorded as a limit). The trial log shows each defect, fix, impact analysis and rerun — material for the evidence dossier (Chapter 48).
46.6 Errors and limits
- Scenarios chosen to pass.
- Rerunning only the failed test after a fix.
- Defects closed without regression tests.
- Results from one run per scenario.
The HVSS-style trial in the worked example was designed against each of these errors. S6 was deliberately adversarial, with a forged tenant claim and conflicting record IDs. After fixing D1, the team reran S1 and AT-01 as well as S2, because the watermark changes R1 latency. Each fix gained a regression test before its defect was closed. Five seeded runs per scenario are why D2 surfaced at all, since it appeared in just one of them.
Exercises — chapter 46
EX091 — Protocol. Objective: write a trial protocol. Inputs: the six scenarios. Steps: write success criteria for S2, S3 and S5 tied to requirement IDs, with the measurement source for each. Expected evidence: criteria table. Rubric: 3 = precise, measurable, sourced; 2 = partial; 1 = vague; 0 = none.
EX092 — Change impact. Objective: analyse impact. Inputs: fix for D2 (per-case serialisation and action idempotency key). Steps: list affected components, interfaces, requirements and tests to rerun, and one new risk the fix introduces. Expected evidence: list. Rubric: 3 = includes throughput impact of serialisation and lock failure behaviour; 2 = partial; 1 = only D2's test; 0 = none.
Chapter synthesis
Trials run pre-specified scenarios with ground truth and instrumentation, classify and prioritise defects, fix with regression tests and change-impact analysis, and iterate until criteria are met or limits are recorded.
Continue studying — chapter 46 of 50
← Chapter 45 · Course chapter index · Chapter 47 →
Continue to the next authored lesson.