Chapter 39 — Capstone evaluation | Bachelor's Quantum and post-quantum security

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 39 — Capstone evaluation

Lesson map

Lesson 39 published. Map, example, EX077.

flowchart TD
  N0["Learning outcomes"]
  N1["39.1 Against the criteria"]
  N2["39.2 Limitations"]
  N3["39.3 Independent review"]
  N4["39.4 Residual uncertainty"]
  N5["39.6 Practical activity"]
  N6["39.8 Learning from the evaluation"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX077"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["39.1 Against the criteria"]
  N2["39.2 Limitations"]
  N3["39.3 Independent review"]
  N4["39.4 Residual uncertainty"]
  N5["39.6 Practical activity"]
  N6["39.8 Learning from the evaluation"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX077"]

[Learning outcomes]
    |
    v
[39.1 Against the criteria]
    |
    v
[39.2 Limitations]
    |
    v
[39.3 Independent review]
    |
    v
[39.4 Residual uncertainty]
    |
    v
[39.6 Practical activity]
    |
    v
[39.8 Learning from the evaluation]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX077]

Learning outcomes

You will (1) evaluate your capstone against its success criteria; (2) analyse limitations and threats to validity; (3) seek and respond to independent review; and (4) state what remains uncertain.

39.1 Against the criteria

Return to the success criteria in your proposal and assess each: met, partly met or not met, with evidence. If criteria changed, explain why. Evaluation is honest when it reports partial success clearly.

For each criterion, point to the evidence: a test run, a figure or a reviewer's comment. A criterion that cannot be assessed should be listed as such with the reason. Examiners respect a clear account of what was and was not achieved more than an evaluation that claims everything worked.

39.2 Limitations

Name limitations by type: data (synthetic, small sample), method (emulated networks), scope (only key exchange), and time (standards status as of a date). For each, state the likely effect on conclusions. Limitations are not apologies; they are the boundary of what your evidence supports.

Be specific about the direction of each effect where you can: for example, emulated networks may understate latency on the worst branch links. Where you cannot say which way a limitation pushes the result, say that too.

39.3 Independent review

Ask a peer or supervisor to rerun part of your work and challenge your conclusions. Record their comments and your responses. Reviewers often find unstated assumptions — for example, that all branches use the same appliance.

Give the reviewer a clear task, such as rerunning one figure from the repository and checking one conclusion. Their result is evidence about reproducibility; their challenge is evidence about the strength of your argument. Report disagreements you could not resolve rather than leaving them out.

39.4 Residual uncertainty

Close with what remains uncertain: quantum timelines, future standards (FN-DSA, HQC), vendor delivery. Explain how a reader should update your conclusions if these change.

For each uncertainty, say which conclusion it affects and in which direction. For example, an earlier quantum timeline would make more flows urgent but would not change which flows lead the ranking. This lets readers apply your work after the assumptions have moved, instead of discarding it. Date the statement, because what counts as uncertain in standards and products changes from year to year.

39.5 Worked example

Evaluation of the CBOM capstone: criterion 1 (reproducible tool) met — a peer reran it and obtained identical output; criterion 2 (stable ranking for z 10–25) partly met — stable for z ≥ 13, changes below; interviews (3) support relevance but are too few to generalise. Limitation: synthetic data may understate unknown assets, which in real inventories are often 20–40 % (an assumption to verify). Reviewer comment: the model ignores partner-controlled segments; response: added as future work with a sketch.

The evaluation reports each finding against the criteria agreed in the proposal.

39.6 Practical activity

Write your evaluation section and ask a peer to rerun one result.

Assess each success criterion with evidence, name the limitations by type and their likely effect, and record what remains uncertain. Give your peer clear instructions and a time limit, record whether they reproduced the result exactly, and note any step that confused them. Fix the instructions where needed, then add their comments and your responses to the evaluation. Keep the evaluation within two pages and put the detailed evidence in an appendix where readers can find it.

39.7 Common errors and limits

Declaring success without evidence per criterion.
Hiding limitations.
Ignoring reviewer findings.
Reporting only the results that support the conclusion.
Treating a limitation as a reason to dismiss the work rather than to bound it.
Calling a criterion met when the evidence is partial.
Asking reviewers who are unlikely to challenge you.
Leaving uncertainties undated.

Each makes the evaluation less useful to the reader who needs to act on it. An independent reviewer helps to catch most of these. Ask for one early.

39.8 Learning from the evaluation

End the evaluation with lessons for the next project: which method choices worked, which assumptions were weakest, and what you would measure first if you started again. These lessons are useful to the organization and to future students, and they show examiners that you understand the work's place in a continuing programme rather than treating it as finished.

Keep the lessons brief and specific. In the CBOM capstone, for example, the weakest assumption was the share of unknown assets, so the first lesson was to measure it early in any real inventory before trusting the ranking.

Exercises

EX077 — Partial success. Write an evaluation sentence for a criterion that was partly met. Rubric: 3 = status, evidence, reason.
EX078 — Limitation effect. For "emulated network instead of real branches", state the likely direction of bias on latency results. Rubric: 3 = underestimates tail latency/middlebox effects, explained.

Synthesis

Evaluation compares results honestly with criteria, names limitations and their effects, incorporates independent review and states residual uncertainty.

Continue studying — chapter 39 of 40

← Chapter 38 (draft) · Course chapter index · Chapter 40 (draft) →