Chapter 39 — Capstone evaluation | Bachelor's Web3, blockchain and cryptoasset security

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 39 — Capstone evaluation

Lesson map

Lesson 39 published. Map, example, EX115.

flowchart TD
  N0["Learning outcomes"]
  N1["39.1 Evaluating against objectives"]
  N2["39.2 User feedback"]
  N3["Synthesis"]
  N0 --> N1 --> N2 --> N3
  N3 --> EX["Worked example"]
  EX --> Q["Exercise EX115"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["39.1 Evaluating against objectives"]
  N2["39.2 User feedback"]
  N3["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> Q["EX115"]

[Learning outcomes]
    |
    v
[39.1 Evaluating against objectives]
    |
    v
[39.2 User feedback]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX115]

Learning outcomes

You will (1) evaluate the capstone against objectives; (2) gather user feedback; (3) identify improvements and limits.

39.1 Evaluating against objectives

Sofia evaluates the monitor against her proposal. It detected all seeded mismatches, meeting the first objective. Alerts arrived within six minutes, meeting the ten-minute target. Weekly routine time fell from thirty to eighteen minutes, a reduction of 40 percent, exceeding the target of a third. She presents each result with its evidence: test logs, alert timestamps and volunteers' routine logs. Measuring against pre-defined objectives makes the evaluation honest, because she cannot redefine success after seeing the results, and the board can check every claim against the stored evidence.

39.2 User feedback

After a month of use, five volunteers rate the monitor. Four find it useful and one finds the email alerts too frequent during the first week, when several late approvals caused alerts. Sofia adjusts the rule so that a mismatch caused by a shift approved within the past 24 hours produces a note in the daily summary rather than an immediate alert. Feedback shows that good technical results do not guarantee a good user experience. She records each piece of feedback and her response, so future maintainers understand why the rules are set the way they are.

39.3 Improvements and limits

Sofia lists improvements for the future: a simple web dashboard, checks on validator health and integration with the cooperative's calendar for keyholder availability. She also states limits: the monitor depends on one validator for data; it cannot detect problems in data that never reach the database; and it was evaluated for only one month. She recommends a second data source and a longer evaluation. Being clear about limits helps the cooperative decide how much to rely on the monitor, and it shows assessors a mature understanding of evaluation.

Worked example

The monitor meets all objectives, cutting routine time 40 percent, and feedback leads to recent-approval mismatches being moved to the daily summary.

Synthetic case. The evaluation and feedback are synthetic.

Errors and limits

Judging success only on technical tests misses whether users find a tool usable.

Exercises

EX115 — Routine reduction. By how much did weekly routine time fall? Rubric: 3 = from thirty to eighteen minutes, 40 percent;; 2 = 40 percent;; 1 = 10 percent;; 0 = none.
EX116 — Rule adjustment. How was the alert rule adjusted? Rubric: 3 = recent-approval mismatches go to the daily summary;; 2 = summary note;; 1 = removed alerts;; 0 = none.
EX117 — Stated limit. Name one limit of the monitor. Rubric: 3 = depends on one validator for data;; 2 = one-month evaluation;; 1 = vague;; 0 = none.

Synthesis

Objective-based evaluation, user feedback and stated limits complete a capstone honestly.

Continue studying — chapter 39 of 40

← Chapter 38 (draft) · Course chapter index · Chapter 40 (draft) →