Chapter 30 — Capstone evaluation and improvement dossier

Chapter 30 — Capstone evaluation and improvement dossier

Lesson map

Lesson 30 published. Map, example, EX088.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["30.1 Evaluation"]
  N2["30.2 Prioritization"]
  N3["30.3 Sustaining the service"]
  N4["Chapter synthesis"]
  N5["Chapter 1: EX001–EX003"]
  N6["Chapter 2: EX004–EX006"]
  N7["Chapter 3: EX007–EX009"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX088"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["30.1 Evaluation"]
  N2["30.2 Prioritization"]
  N3["30.3 Sustaining the service"]
  N4["Chapter synthesis"]
  N5["Chapter 1: EX001–EX003"]
  N6["Chapter 2: EX004–EX006"]
  N7["Chapter 3: EX007–EX009"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX088"]

[Learning outcomes and connection]
    |
    v
[30.1 Evaluation]
    |
    v
[30.2 Prioritization]
    |
    v
[30.3 Sustaining the service]
    |
    v
[Chapter synthesis]
    |
    v
[Chapter 1: EX001–EX003]
    |
    v
[Chapter 2: EX004–EX006]
    |
    v
[Chapter 3: EX007–EX009]
    |
    v
[Worked example] --> [Exercise EX088]

Learning outcomes and connection

You will be able to (1) evaluate the capstone service against its criteria and measures honestly; (2) prioritize improvements with owners, effort and expected effect; and (3) assemble an improvement dossier that hands the service into steady operation. This final chapter closes the course and the service's first build cycle.

30.1 Evaluation

Combine three kinds of evidence: acceptance results (Chapter 29), service measures over a period (time to triage, stage latencies, record completeness, detection outcome labels — Chapters 9, 19, 20, 23) and exercise findings (Chapter 28). For each component, state: met / partly met / not met, the evidence, and confidence (how much data, how representative). Report limitations: small case counts, test-environment demonstrations, observer effects. Fernmoor's first-year dossier applies this to five areas, from evidence flows to supplier coordination, and marks triage latency as moderate confidence because it rests on 19 out-of-hours cases.

30.2 Prioritization

List improvement items from all sources. For each: problem with evidence, proposed change, owner, effort (S/M/L), expected effect on a named measure, and dependencies. Prioritize with a simple, explicit rule — for example, first items that protect recovery or evidence integrity, then items with large expected effect and small effort. NIST SP 800-61 Rev. 3 emphasizes that lessons learned feed improvement across all CSF functions; the dossier is that feedback made concrete. Fernmoor's dossier put the untested permits share restore first because it protects recovery, followed by loading supplier contacts, a second on-call and phishing-resistant authenticators.

30.3 Sustaining the service

A response service decays without maintenance: owners move, flows break, playbooks go stale. Build a sustainment calendar (monthly measures and checks, quarterly action tests and restore tests, semi-annual exercises, annual charter review), ownership for every component, and a backlog process. Blameless post-incident reviews (as in the SRE book's postmortem culture and Dekker's just culture) keep people reporting problems. Fernmoor's dossier closes with exactly this calendar and an owner list, so the service has a schedule for the year ahead and a person for each check.

30.4 Worked example (synthetic): Fernmoor's first-year dossier

Fictional. Excerpt of the evaluation:

Top improvements (priority order): (1) restore test for permits share — protects recovery; owner Infrastructure; S; measure: tested RTO/RPO. (2) load supplier contacts into inventory — owner Procurement; M; measure: exercise objective 5. (3) second on-call for out-of-hours peaks — owner IT director; M; measure: out-of-hours p90 ≤ 30 min over two quarters. (4) phishing-resistant authenticators for staff with resident-data access — owner Identity; L; measure: share of such staff with security keys, and identity-case count.

The dossier closes with the sustainment calendar, the owner list, open risks formally accepted by the director (OT scope still excluded; referral procedure in place), and a list of what the evaluation could not show.

30.5 Common errors and limits

• Reporting only successes.
• Improvements without measures, so no one knows whether they worked.
• No sustainment plan; the service degrades within a year.
• Confidence claims that ignore small samples.

Fernmoor's dossier guards against each. It lists supplier coordination as not met, gives a measure for every improvement, includes the sustainment calendar, and marks the out-of-hours triage figure as moderate confidence because it rests on 19 cases. It also lists what the evaluation could not show, such as the restore time of the untested permits share. Each of these makes the dossier more useful than a list of successes.

Exercises — chapter 30

Attempt each exercise before opening the worked results. Every case, organization, person, address and number in an exercise is synthetic unless a cited public source is named.

EX088 — Evaluation row. Steps: write an evaluation row for "detection register" using Chapter 19 data. Rubric: 3 = status, evidence with figures, confidence with reason, limitation; 2 = missing confidence; 1 = assertion; 0 = none.

EX089 — Prioritize. Steps: rank five improvement items (permits restore test; supplier contacts; second on-call; security keys; mobile runbook fix) with your rule and justify. Rubric: 3 = explicit rule applied consistently with effort/effect; 2 = ranking with partial reasons; 1 = list; 0 = none.

EX090 — Sustainment calendar. Steps: write a twelve-month sustainment calendar. Rubric: 3 = monthly, quarterly, semi-annual and annual activities with owners covering measures, checks, action/restore tests, exercises, charter and inventory review; 2 = missing owners; 1 = partial; 0 = none.

Chapter synthesis

Evaluate the service with acceptance results, measures and exercise findings, stating confidence and limits; prioritize improvements by an explicit rule with owners and measures; and hand over a sustainment calendar so the service keeps working after the course ends.

Continue studying — chapter 30 of 30

← Chapter 29 · Course chapter index · Next course via chapter index →

This is the final chapter of this course; the index links the next course in the full study path.