Chapter 39 — Capstone evaluation

Chapter 39 — Capstone evaluation

Lesson map

Lesson 39 published. Map, example, EX115.

flowchart TD
  N0["Learning outcomes"]
  N1["39.1 Evaluation results"]
  N2["39.3 Next steps"]
  N3["Synthesis"]
  N0 --> N1 --> N2 --> N3
  N3 --> EX["Worked example"]
  EX --> Q["Exercise EX115"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["39.1 Evaluation results"]
  N2["39.3 Next steps"]
  N3["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> Q["EX115"]

[Learning outcomes]
    |
    v
[39.1 Evaluation results]
    |
    v
[39.3 Next steps]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX115]

Learning outcomes

You will (1) evaluate the capstone against its criteria; (2) measure adoption and false alerts; (3) recommend next steps.

39.1 Evaluation results

Priya evaluates the capstone against the proposal. In the seeded test environment, discovery finds fifteen of fifteen certificates. In production, it finds forty-two certificates, including three that nobody had recorded. Over two months, alerts fire for six real expiries, all renewed in time, with one false alert in total. Staff complete common inventory tasks after a twenty-minute training. The aim is met for the scoped systems. She presents these results in a table that compares each measured value with the target set in the proposal, showing exactly where the aim was met.

39.2 Adoption and limits

Adoption is uneven: two branch IT volunteers ignore alert messages sent to personal addresses, so Priya routes alerts to the shared mailbox monitored by Grace's team. She lists limits: the inventory covers only scanned hosts, cannot see certificates on devices outside the network and depends on the staff directory being current. She reports that one alert was acted on only after the seven-day reminder, showing the second alert's value. She adds a monthly check that the staff directory matches current owners.

39.3 Next steps

Priya recommends next steps: add code-signing keys, as Grace requested; integrate with ACME renewal logs to spot failed renewals sooner; and review owners quarterly. She also recommends that the consortium keep the tool simple and maintained rather than adding features that would require skills the team lacks. Her evaluation report includes data tables, screenshots and a short guide for maintainers. Grace agrees to assign one team member as the tool's maintainer, with time set aside each month. Priya offers to help during the first maintenance cycle so that knowledge transfers smoothly.

Worked example

Priya's evaluation finds forty-two production certificates including three unrecorded, six expiries handled with one false alert, and recommends routing alerts to a shared mailbox and modest next steps.

Synthetic case. Results are invented.

Errors and limits

Two months is short for evaluating reliability. Tools without maintainers decay.

Exercises

• EX115 — Production findings. How many certificates were found in production, and how many were unrecorded? Rubric: 3 = 42, including 3 unrecorded;; 2 = one number;; 1 = wrong;; 0 = none.
• EX116 — Alert routing. Why were alerts moved to a shared mailbox? Rubric: 3 = volunteers ignored alerts sent to personal addresses;; 2 = ignored;; 1 = preference;; 0 = none.
• EX117 — Second alert value. What showed the value of the seven-day reminder? Rubric: 3 = one alert was acted on only after the seven-day reminder;; 2 = reminder;; 1 = vague;; 0 = none.

Synthesis

Honest evaluation measures results, adoption and limits, and leads to modest, sustainable next steps.

Continue studying — chapter 39 of 40

← Chapter 38 (draft) · Course chapter index · Chapter 40 (draft) →