Chapter 39 — Capstone evaluation

Lesson map

Lesson 39 published. Map, example, EX115.

flowchart TD
  N0["Learning outcomes"]
  N1["39.1 Evaluation data"]
  N2["39.2 Analysing the samples"]
  N3["Lab — Knowing how to report, before and after"]
  N4["39.3 Honest reporting"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX115"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["39.1 Evaluation data"]
  N2["39.2 Analysing the samples"]
  N3["Lab — Knowing how to report, before and after"]
  N4["39.3 Honest reporting"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX115"]

[Learning outcomes]
    |
    v
[39.1 Evaluation data]
    |
    v
[39.2 Analysing the samples]
    |
    v
[Lab — Knowing how to report, before and after]
    |
    v
[39.3 Honest reporting]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX115]

Learning outcomes

You will (1) evaluate the capstone against criteria; (2) analyse small before and after samples; (3) report honestly.

39.1 Evaluation data

Nadia compares three months before and three months after the guide. Before, volunteers and part-time staff made 4 reports; after, 15. She also surveys them, receiving 34 responses before and 37 after, asking whether they would know how to report a suspicious message and whether they would worry about being blamed. She notes that different people answered each survey, since volunteers change, so the comparison is between groups rather than individuals. The survey takes three minutes, is anonymous and asks no questions about personal details, so volunteers can answer quickly at the end of a session.

39.2 Analysing the samples

The lab computes the share answering yes to knowing how to report before and after, with Wilson intervals. Knowledge rose clearly. Worry about blame fell, though intervals overlapped. Reports rose from 4 to 15, but with small counts and possible seasonal effects, such as more sessions in autumn, she treats the rise as encouraging rather than conclusive. All three success criteria are met, so the predefined revision rule is not triggered and the programme continues. Knowledge rose from 41% to 81%. Worry fell from 35% to 19%.

Lab — Knowing how to report, before and after

# Knowing how to report: before and after the guide, Wilson intervals (synthetic).
from math import sqrt
def wilson(k, n, z=1.96):
    p = k / n; d = 1 + z * z / n; c = (p + z * z / (2 * n)) / d; h = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return c - h, c + h
for name, k, n in (("know before", 14, 34), ("know after", 30, 37), ("worry before", 12, 34), ("worry after", 7, 37)):
    lo, hi = wilson(k, n); print(f"{name}: {k}/{n} = {k / n:.0%} (95% CI {lo:.0%} to {hi:.0%})")

Run with python3 lab_capstone.py. Verified output:

know before: 14/34 = 41% (95% CI 26% to 58%)
know after: 30/37 = 81% (95% CI 66% to 91%)
worry before: 12/34 = 35% (95% CI 21% to 52%)
worry after: 7/37 = 19% (95% CI 9% to 34%)

39.3 Honest reporting

Nadia writes her evaluation report with results, limits and recommendations. She notes that the survey groups differ, that seasonal effects may explain part of the change and that knowing how to report is not the same as reporting. She recommends keeping the guide and form, adding a short annual reminder and checking again in a year. Dr Prescott praises the report for neither overstating nor dismissing the results, which is the balance good evaluation requires. Managers accept every recommendation. The report is shared with staff.

Worked example

Nadia evaluates her capstone, finds knowledge rising from 41% to 81% and reports increasing from 4 to 15, and reports limits honestly.

Synthetic case. Data are synthetic.

Errors and limits

Different respondents before and after. Seasonal effects.

Exercises

• EX115 — Report counts. How did volunteer and part-time reports change?
  Show marking criteria

  Rubric: 3 = from 4 to 15;; 2 = rise;; 1 = vague;; 0 = none.
• EX116 — Knowledge shares. What shares knew how to report before and after in the lab?
  Show marking criteria

  Rubric: 3 = 41% and 81%;; 2 = rise;; 1 = vague;; 0 = none.
• EX117 — Groups caution. Why is the survey comparison between groups?
  Show marking criteria

  Rubric: 3 = volunteers changed between surveys;; 2 = turnover;; 1 = vague;; 0 = none.

Synthesis

Evaluations compare against criteria, analyse small samples carefully and report limits.

Continue studying — chapter 39 of 40

← Chapter 38 (draft) · Course chapter index · Chapter 40 (draft) →