Chapter 39 — Capstone evaluation

Chapter 39 — Capstone evaluation

Lesson map

Lesson 39 published. Map, example, EX077.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["39.1 Against the criteria you set"]
  N2["39.2 Sensitivity and robustness"]
  N3["39.3 Limitations with direction"]
  N4["39.4 Independent review"]
  N5["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX077"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["39.1 Against the criteria you set"]
  N2["39.2 Sensitivity and robustness"]
  N3["39.3 Limitations with direction"]
  N4["39.4 Independent review"]
  N5["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX077"]

[Learning outcomes and connection]
    |
    v
[39.1 Against the criteria you set]
    |
    v
[39.2 Sensitivity and robustness]
    |
    v
[39.3 Limitations with direction]
    |
    v
[39.4 Independent review]
    |
    v
[Chapter synthesis]
    |
    v
[Worked example] --> [Exercise EX077]

Learning outcomes and connection

After this chapter you can (1) evaluate your capstone's results against the success criteria fixed in the proposal; (2) carry out sensitivity and robustness checks on your comparison; (3) state limitations specifically, with their likely direction of effect; and (4) seek and respond to independent review before the final thesis. It builds on Chapters 34, 37 and 38 and prepares the thesis and presentation (Chapter 40).

39.1 Against the criteria you set

Evaluation begins with the proposal's success criteria (Chapter 35). Were they met, partly met or not met? Report each honestly with evidence. Then evaluate the comparison itself: are the scores supported by the evidence register; would another person, using your anchored scales, give similar scores (ask one); did any constraint exclude an option? In the worked evaluation (§39.5), one interview took place and the second was replaced by documents, so the practitioner-input criterion is reported as partly met rather than quietly dropped. Asking a second person to score two options with your anchored scales is a quick reliability check; if their scores differ by more than one point on any criterion, your anchors need clearer wording before the thesis.

39.2 Sensitivity and robustness

Re-run your comparison under plausible variations: different weights (Lab 13), pessimistic and optimistic evidence (Chapter 22's halved autonomy), alternative model assumptions (Chapter 38's archive variant). A robust recommendation holds across most variations; a fragile one depends on a specific assumption, which you must name. Present the result as a short table: variation, winner, comment. The worked evaluation shows a fragile result: the gateway wins once cost weight exceeds 0.45, and accelerated replacement wins when isolation's monitoring is assumed not to work. Both are named in the conclusion as reversal conditions, so a reader can see which value judgement and which assumption the recommendation rests on.

39.3 Limitations with direction

"The sample was small" is a weak limitation. A useful limitation says what, why it matters and in which direction it probably biases the result. For example: "The exposure model assumes every unsupported device is equally exploitable; in practice internet-adjacent devices are more exposed, so the model probably understates the benefit of isolation for imaging devices." Directional limitations let readers adjust your conclusions rather than discard them. Where the direction is genuinely unknown, as with the synthetic inventory, say so explicitly rather than leaving the limitation undirected by omission.

39.4 Independent review

Before the thesis is final, ask someone outside your project — a peer from the seminar (Chapter 37), your host supervisor, an academic not involved — to review the evaluation chapter. Record their points and your responses. Independence is the weakest link in most evidence (Chapter 15); the capstone is where you practise strengthening it. In the worked evaluation the reviewer's point about unstaffed monitoring changed the main table: the pessimistic variation became a headline result, and the recommendation became conditional. Record the reviewer's role and independence beside each point, so an assessor can judge how much weight the review carries.

39.5 Worked example: evaluation table (synthetic)

This evaluation is synthetic. Unsupported-devices capstone:

Limitations (directional): synthetic inventory (direction unknown — stated as such); one practitioner interview (probably understates clinical disruption of isolation, so isolation's score may be optimistic); exposure model ignores attacker targeting (probably understates the value of removing the most exposed devices first). Reviewer's main point: "isolation's advantage depends on monitoring being staffed; show what happens if alerts are not reviewed." Response: the pessimistic variation addresses this and is now in the main table.

39.6 Common errors and limits

• Moving the goalposts: evaluating against criteria invented after results.
• Sensitivity analysis on irrelevant parameters only.
• Limitations without direction.
• Reviews from people who helped produce the work.

The worked evaluation avoids the first by using the proposal's criteria, varies the weights that matter, gives each limitation a direction, and uses a reviewer who did not help produce the work. The second error is common: varying the evidence-strength weight, which barely changes the ranking, while leaving cost untouched, which flips it. A limitation such as "one interview" with no direction leaves the reader unable to adjust the result. A reviewer who helped build the exposure model cannot provide independent review of it.

Exercises — chapter 39

EX077 — Directional limitations. Objective: write useful limitations. Inputs: the Chapter 34 comparison (Lab 13). Steps: write three limitations, each with what, why and probable direction. Expected evidence: three limitations. Rubric: 3 = specific (e.g. scores by one assessor — direction unknown, raise via second scorer; backups' score ignores data theft — overstates its benefit against double-extortion ransomware; cost scores exclude staff time for drills — overstates backups' cost advantage); 2 = without direction; 1 = generic; 0 = none.

EX078 — Respond to review. Objective: handle critique. Inputs: reviewer says "your restore-time evidence is from one test environment and five drills". Steps: write a response that accepts, partially accepts or declines with reasons, and states any change. Expected evidence: response. Rubric: 3 = accepts the limitation, adds Wilson interval (Chapter 37), states the test–production differences and the direction of bias, adjusts confidence in the recommendation; 2 = accepts without change; 1 = defensive; 0 = none.

Chapter synthesis

Evaluate against the criteria you set before the results, test the comparison's robustness, state limitations with their direction and invite independent review. A fragile recommendation honestly labelled is more useful than a robust-looking one that hides its assumption.

Continue studying — chapter 39 of 40

← Chapter 38 · Course chapter index · Chapter 40 →

Continue to the next authored lesson.