Chapter 39 — Capstone evaluation
Lesson map
Lesson 39 published. Map, example, EX115.
flowchart TD
N0["Learning outcomes"]
N1["39.1 Adoption"]
N2["39.2 Secrets"]
N3["39.3 Setup time"]
N4["39.4 Feedback"]
N5["39.6 Handover"]
N6["Qualitative evidence"]
N7["Ongoing measures"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX115"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["39.1 Adoption"]
N2["39.2 Secrets"]
N3["39.3 Setup time"]
N4["39.4 Feedback"]
N5["39.6 Handover"]
N6["Qualitative evidence"]
N7["Ongoing measures"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX115"]
[Learning outcomes]
|
v
[39.1 Adoption]
|
v
[39.2 Secrets]
|
v
[39.3 Setup time]
|
v
[39.4 Feedback]
|
v
[39.6 Handover]
|
v
[Qualitative evidence]
|
v
[Ongoing measures]
|
v
[Worked example] --> [Exercise EX115]
Learning outcomes
You will (1) evaluate the capstone against success criteria; (2) collect quantitative and qualitative evidence; (3) identify limitations; (4) plan handover.
39.1 Adoption
Five projects adopted the template in the term, above the target of four.
The five projects include the two pilot teams and three that joined after the workshop. Lena reports how each heard about the template, which suggests the workshop drove most adoption. She notes that adoption counts projects that completed setup, not those that downloaded the template, since the second figure would be larger but less meaningful. All five had completed a working first build with every check enabled, which Lena confirmed from their CI records.
39.2 Secrets
No secrets reached public repositories in adopting projects; the scanner caught two before push.
The criterion is met, but the term is short and the number of projects small, so Lena does not claim the template prevents all leaks. The two caught secrets show that the hook works in other projects as it did at CampusEats. Both developers fixed the problem before pushing, and neither secret needed rotation. Lena confirmed this from the scanner logs of each adopting project, with the teams' permission.
39.3 Setup time
Median setup 40 minutes across five projects.
The median is well within the one-hour criterion in the proposal. Lena reports every project's setup time alongside it, since a median from five values can hide one slow case, and she notes where the time went: most of it was spent reading the README and creating the signing identity with the new setup script. Teams whose CI service needed extra permissions took longest, so the README now lists those permissions at the start. The measure counts only setup, not the time teams later spent learning to read the results
39.4 Feedback
Teams liked ready-made tests; some found SBOM output confusing. Lena adds a short guide.
Feedback came from a short survey and a conversation with each team. The ready-made tests saved teams from writing basic session and header checks themselves. The SBOM confusion came from long output that nobody had explained; the guide shows how to search it during an advisory, which is how CampusEats uses it. Lena lists each request and whether she acted on it. Teams also asked for help with update pull requests.
39.5 Limits
One term, five projects, volunteers who may have been more security-minded than average.
Lena also notes that adopting projects were mostly new, so the template's effect on established projects with existing habits is unknown. She cannot separate the template's effect from the workshop's, since every adopting team attended it. Each limit is stated with its likely effect, and she suggests a follow-up study with older projects and teams that did not attend the workshop. The evaluation also states the dates covered.
39.6 Handover
Grace's office takes ownership with a named maintainer; the monthly update job runs automatically.
The maintainer has access to the repository and a handover note covering the release process, the scanner allow list and the support questions received so far. Lena stays available as a contributor. Handover before graduation meets the risk in the proposal about maintenance after she leaves, and Grace's office adds the template to its annual review. The note is stored in the template's repository, where any future maintainer can find it.
Qualitative evidence
Lena interviews one member of each adopting team for fifteen minutes. Common themes: ready-made tests saved time, the update bot reduced worry about old packages, and the setup script made signing painless. One team said the checks felt strict at first but helped them explain security to their users. Interviews explain why the numbers look as they do and suggest improvements, such as clearer messages when the advisory check fails. Lena codes the answers into themes with a classmate and they agree on all but one, which they discuss and resolve.
Ongoing measures
The template's maintainer will report adoption, secrets caught and setup time each term, using the same measures, so the evaluation continues after Lena leaves and later results can be compared fairly.
Reporting
Lena writes a two-page evaluation report for Grace and the project leads, with measures, quotes and recommendations.
Worked example
Lena compares adopting projects with three similar non-adopting ones: the non-adopters had one secret reach a public repository and a median of 20 days to fix high advisories, versus 5 for adopters. She notes that the groups differ in other ways and that the comparison is indicative only.
Synthetic case. Results are synthetic.
Errors and limits
Success in the first term may fade without continued support.
Exercises
- EX115 — Criteria results. Report results against the three criteria. Rubric: 3 = 5 projects; zero secrets; 40 min; 2 = two; 1 = one; 0 = none.
- EX116 — Feedback. What feedback led to a change? Rubric: 3 = SBOM output confusing; guide added; 2 = partial; 1 = vague; 0 = none.
- EX117 — Comparison caution. Why is the adopter comparison only indicative? Rubric: 3 = groups differ in other ways; 2 = partial; 1 = vague; 0 = none.
Synthesis
Evaluate against criteria with honest limits and hand over to a lasting owner.
Continue studying — chapter 39 of 40
← Chapter 38 (draft) · Course chapter index · Chapter 40 (draft) →