Chapter 39 — Capstone evaluation
Lesson map
Lesson 39 published. Map, example, EX115.
flowchart TD
N0["Learning outcomes"]
N1["39.1 Against the proposal"]
N2["39.2 Results versus interpretation"]
N3["39.3 Self-assessment rubric"]
N4["39.4 Successor plan"]
N5["39.5 The supervisor's view"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX115"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["39.1 Against the proposal"]
N2["39.2 Results versus interpretation"]
N3["39.3 Self-assessment rubric"]
N4["39.4 Successor plan"]
N5["39.5 The supervisor's view"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX115"]
[Learning outcomes]
|
v
[39.1 Against the proposal]
|
v
[39.2 Results versus interpretation]
|
v
[39.3 Self-assessment rubric]
|
v
[39.4 Successor plan]
|
v
[39.5 The supervisor's view]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX115]
Learning outcomes
You will (1) evaluate a capstone against its own proposal; (2) separate results, interpretation and limitations; (3) run a structured self-assessment against the course rubric; and (4) plan the work a successor would need to do.
39.1 Against the proposal
Return to the acceptance criteria written in Chapter 35. For each, record met, partly met or not met, with evidence. A capstone that misses a criterion but explains why — for example, the drift class could not be detected by rules alone — can still earn a strong grade.
Maya's evaluation table shows the three possible outcomes in use. Criteria 1 and 2 are met, with the confusion matrix and the denominators as evidence, while criterion 3 is partly met: five of six attack classes after the cumulative rule, with the HMI-originated insider write still missed and the reason stated.
39.2 Results versus interpretation
Results are what the evaluation produced: "recall 4/5 on the eight-case set". Interpretation is what you think it means: "the cumulative rule addresses slow manipulation". Limitations bound the interpretation: "tested only on synthetic data with one drift profile". Keeping three headings prevents results from silently absorbing opinion.
In Maya's thesis, recall rose to 4/5 on the eight-case set once the cumulative-change rule caught the drift attack; the HMI-1 write remains the miss. That sentence is a result. "The monitor is suitable for LDHP" would be an interpretation that the limitation, one drift profile on synthetic data, does not yet support.
39.3 Self-assessment rubric
Score yourself 0–3 on: problem framing, method fit, evidence quality, safety and ethics, communication. Write one sentence of evidence per score. Supervisors use the same rubric; differences start a useful conversation.
Maya scored herself 2 on evidence quality, with the sentence "eight cases is a toy set", and 3 on safety and ethics, with "offline only, documented". The worked results explain why 2 rather than 3 is fair: the analysis is reproducible and reports denominators, but eight synthetic cases with one drift profile cannot support general claims. A self-score of 3 there would have been the first point of disagreement with her supervisor.
39.4 Successor plan
List what you would do with three more months: a second process model, real (owner-provided) anonymised captures, an operator usability study. Rank by value to the question.
For Maya, the ranking puts the HMI-originated write first, because it is the class her monitor still misses: correlating writes with operator logins on owner-provided, anonymised captures would test the remedy. A second process model, such as the expansion vessel from Chapter 21, comes next, since a monitor tuned to one loop may not transfer. An operator usability study comes last, because it matters only once the monitor detects the classes that matter.
Worked example (synthetic LDHP)
Maya's evaluation table: criterion 1 (processes 8-case set, reports confusion matrix) — met; criterion 2 (precision and recall with denominators) — met; criterion 3 (detects all six attack classes) — partly met, five of six after the cumulative rule, HMI-originated insider write still missed. Self-assessment: evidence quality 2 ("eight cases is a toy set"), safety 3 ("offline only, documented").
Synthetic case — Lakeside District Heating Plant. Scores are a worked illustration.
39.5 The supervisor's view
Supervisors look for alignment between the question, the evidence and the claims. A frequent comment at this stage is that the conclusion section repeats the results instead of answering the research question. Rewrite the conclusion so its first sentence answers the question from Chapter 35 directly, its second states the strongest limitation, and its third names the next study.
Maya's first sentence would answer the question from her proposal directly: on synthetic LDHP captures, the rule-based monitor classified five of six attack classes correctly.
Errors and limits
Self-assessment tends to over-score communication and under-score method fit. Do not re-define acceptance criteria after the fact to fit results.
Exercises
- EX115 — Classifying three sentences. Label as result, interpretation or limitation: (a) "precision 4/5"; (b) "the tool is ready for pilot"; (c) "one drift profile tested". Rubric: 3 = R, I, L; 2 = two; 1 = one; 0 = none.
- EX116 — Scoring evidence quality. Justify a score of 2 (not 3) for evidence quality in Maya's capstone. Rubric: 3 = reproducible and correct but small synthetic set limits generality; 2 = partial; 1 = "could be better"; 0 = none.
- EX117 — The top successor task. Which successor task most strengthens the research question and why? Rubric: 3 = additional process model or owner-provided captures to test generalisation; 2 = task only; 1 = UI polish; 0 = none.
Synthesis
Evaluate a capstone against its own promises, keep results apart from opinions, and leave a ranked path for whoever continues the work.
Continue studying — chapter 39 of 40
← Chapter 38 (draft) · Course chapter index · Chapter 40 (draft) →