Chapter 39 — Capstone evaluation against explicit criteria
Lesson map
Lesson 39 published. Map, example, EX115.
flowchart TD
N0["Learning outcomes and connection"]
N1["39.1 Criteria first"]
N2["39.2 Baseline"]
N3["39.3 Error analysis"]
N4["39.4 Repeatability"]
N5["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX115"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["39.1 Criteria first"]
N2["39.2 Baseline"]
N3["39.3 Error analysis"]
N4["39.4 Repeatability"]
N5["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX115"]
[Learning outcomes and connection]
|
v
[39.1 Criteria first]
|
v
[39.2 Baseline]
|
v
[39.3 Error analysis]
|
v
[39.4 Repeatability]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX115]
Learning outcomes and connection
You will be able to (1) evaluate a capstone against pre-specified acceptance criteria and a baseline; (2) analyse errors by type and cause; (3) check repeatability; and (4) write limitations that say where the workflow did not meet its aims. Chapter 35 set the criteria; Chapter 38 built the workflow. Chapter 40 turns the evaluation into a thesis.
39.1 Criteria first
Evaluate against the criteria written in the proposal, not new ones chosen after seeing results. If a criterion proved poorly defined, report it as defined, then add a clearly labelled post-hoc analysis.
The run recorded in Lab 39 differs from the proposal in one respect that the thesis must report. The proposal planned twelve synthetic scenarios, with each volunteer reporting six by email and six by form; the evaluated run uses eight cases, T01 to T08, each reported once by email and once by form. A deviation like this is acceptable only if it is stated and explained, and it never licenses changing the criteria. Here the criteria stay at 90 per cent for completeness and routing.
39.2 Baseline
Compare with the baseline (the old process, or the workflow without the change). Without a baseline, "90% complete" has no meaning.
In Lab 39 the baseline is the email process run on the same eight cases, which shows four of eight reports complete and six of eight routed correctly. Against that, the form's seven of eight on both measures is a clear improvement even though it falls short of the threshold. Reporting the baseline and the criterion side by side lets a reader see both facts at once.
39.3 Error analysis
Classify failures: wrong routing, missing field, unauthorized action allowed, legitimate action denied, handoff lost. For each class, find the cause: design, configuration, test case, participant misunderstanding. Error analysis is often more valuable than the headline number.
Lab 39 shows the value: the form failed on two cases for different reasons. T03 exposed a missing routing rule, a design gap; T06 exposed a reporter writing system details into a free-text field that overflowed, a design and usability issue. Each cause suggests a specific fix that can be tested. The email failures, by contrast, cluster around missing system and contact details, which is exactly what the form's required fields were meant to address.
39.4 Repeatability
Rerun the test suite with a fresh environment and, if possible, a different operator. Report whether results match exactly and explain differences.
For Lab 39, a repeat run after fixing the T03 routing rule and the T06 field should use new held-out cases as well as the original eight, reported separately, so improvement on the fixed cases is not mistaken for general improvement. With only eight cases per method, a single case moves a rate by twelve and a half percentage points, so small differences between runs are expected. Record the operator and environment for each run.
39.5 Worked example and lab (synthetic): handoff tests
Capstone records from a fictional run of the Chapter 35 intake workflow (form) versus baseline (email). Save as lab39.py.
# Lab 39 - evaluate capstone test records against pre-specified criteria (synthetic, stdlib only)
from statistics import median
# (case, method, complete, routed_correctly, minutes_to_triage, error_class)
RUNS = [
("T01","email",0,1,14,"missing system"), ("T01","form",1,1,6,""),
("T02","email",1,1, 9,""), ("T02","form",1,1,5,""),
("T03","email",0,0,22,"wrong queue"), ("T03","form",1,0,7,"routing rule gap"),
("T04","email",1,1, 8,""), ("T04","form",1,1,4,""),
("T05","email",0,1,16,"missing contact"), ("T05","form",1,1,6,""),
("T06","email",1,0,12,"wrong queue"), ("T06","form",0,1,9,"free-text overflow"),
("T07","email",0,1,15,"missing system"), ("T07","form",1,1,5,""),
("T08","email",1,1,10,""), ("T08","form",1,1,6,""),
]
CRITERIA = {"complete": 0.90, "routed": 0.90}
for m in ("email", "form"):
r = [x for x in RUNS if x[1] == m]
comp = sum(x[2] for x in r) / len(r)
rout = sum(x[3] for x in r) / len(r)
med = median(x[4] for x in r)
ok = comp >= CRITERIA["complete"] and rout >= CRITERIA["routed"]
print(f"{m:5}: complete {comp:.3f} routed {rout:.3f} median triage {med} min -> criteria met: {ok}")
errors = {}
for x in RUNS:
if x[5]:
errors.setdefault(x[1], []).append(f"{x[0]}:{x[5]}")
for m, e in errors.items():
print(m, "errors:", e)
Expected output:
email: complete 0.500 routed 0.750 median triage 13.0 min -> criteria met: False
form : complete 0.875 routed 0.875 median triage 6.0 min -> criteria met: False
email errors: ['T01:missing system', 'T03:wrong queue', 'T05:missing contact', 'T06:wrong queue', 'T07:missing system']
form errors: ['T03:routing rule gap', 'T06:free-text overflow']
Interpretation: the form greatly improves completeness (0.50 → 0.875) and halves median time to triage (13 → 6 minutes), but it does not meet the pre-specified 90% thresholds for completeness or routing. Error analysis identifies two specific causes: a routing rule missing for T03's case type, and T06's reporter writing system details into a free-text field that overflowed. Both are fixable, but the honest result is "criteria not met in this run". Note also: 8 cases per arm is small; one case is 12.5 percentage points. Repeatability: rerun after fixes with the same cases would test the fixes but risks tuning to the test set; add new cases for the re-evaluation.
39.6 Common errors and limits
- Changing criteria after the fact.
- Reporting only the headline improvement.
- Ignoring the size of the test set.
-
Re-testing on the same cases after tuning and calling it validation.
-
Hiding a change between the planned and the actual test set, such as running eight cases where twelve were proposed, instead of reporting and explaining it.
- Counting a fixed error as a pass in the original run.
- Describing the form's seven of eight as meeting the 90 per cent threshold by rounding or by excluding a difficult case.
- Reporting medians without the number of cases behind them.
Exercises — chapter 39
Attempt each exercise before opening the worked results. Every case, organization, person, address and number in an exercise is synthetic unless a cited public source is named.
EX115 — Run and state. Steps: run Lab 39 and write the results paragraph in calibrated language. Rubric: 3 = improvements quantified, criteria not met stated plainly, error causes named, sample size noted; 2 = omits unmet criteria; 1 = overclaims; 0 = none.
EX116 — Fix and re-evaluate plan. Steps: propose fixes for the two form errors and a re-evaluation design that avoids tuning to the test set. Rubric: 3 = add routing rule for T03 type, structured field or length validation for T06; new held-out cases plus original cases reported separately; 2 = fixes only; 1 = re-run same set; 0 = none.
EX117 — Limitations. Steps: write four limitations with the likely direction of bias. Rubric: 3 = small n (imprecise either way), synthetic cases designed by the author (may favour the form), volunteers not under clinical pressure (may overstate form usability), single run; directions stated; 2 = list without direction; 1 = generic; 0 = none.
Chapter synthesis
Evaluate against the criteria you fixed in advance, compare with a baseline, analyse every error for its cause, test repeatability on fresh cases and report unmet aims plainly.
Continue studying — chapter 39 of 40
← Chapter 38 · Course chapter index · Chapter 40 →
Continue to the next authored lesson.