Chapter 39 — Capstone evaluation
Lesson map
Lesson 39 published. Map, example, EX115.
flowchart TD
N0["Learning outcomes"]
N1["39.1 Criteria results"]
N2["39.2 Trend"]
N3["39.3 Feedback"]
N4["39.5 Independent check"]
N5["39.6 What next"]
N6["Measuring user value"]
N7["Survey"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX115"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["39.1 Criteria results"]
N2["39.2 Trend"]
N3["39.3 Feedback"]
N4["39.5 Independent check"]
N5["39.6 What next"]
N6["Measuring user value"]
N7["Survey"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX115"]
[Learning outcomes]
|
v
[39.1 Criteria results]
|
v
[39.2 Trend]
|
v
[39.3 Feedback]
|
v
[39.5 Independent check]
|
v
[39.6 What next]
|
v
[Measuring user value]
|
v
[Survey]
|
v
[Worked example] --> [Exercise EX115]
Learning outcomes
You will (1) evaluate the capstone against success criteria; (2) measure outcomes; (3) gather user feedback; (4) state limits honestly.
39.1 Criteria results
All five check families run weekly (met). Anonymous repositories: 0 (met). Logging coverage 6 of 6 (met). Median findings age 9 days at term end (met). Scan blocking key-like strings (met, one real key blocked).
Every criterion was met, and each has evidence in the dossier: the weekly reports, the registry export showing no anonymous repositories, the logging coverage check, the findings ages and the scan log showing the blocked key. Samuel presents them in a table with the target, the result and the evidence location.
39.2 Trend
Total open findings fell from 41 at the start of term to 9.
The fall came from owners fixing findings, not from checks being removed: the check families stayed the same throughout. Samuel notes that new findings still appear each week, and the trend shows they are being fixed faster than they arrive. He shows the weekly totals on a chart with the start and end marked. The steepest fall came in the first month, when the anonymous repositories, the missing logging and several stale images were fixed together. After that, the total fell more slowly as remaining findings needed more work, such as rebuilding images with non-root users.
39.3 Feedback
The cloud team rates the report useful; they ask for a shorter summary at the top, which Samuel adds.
The request for a shorter summary showed how the report is actually read: quickly, on Monday mornings. Samuel's original version listed all findings first, which buried the important ones. The new summary lists only new and high-severity findings, with the full list below. Samuel asked two follow-up questions: which parts were never read, and which were missing. The answers led to the runbook links suggested in the survey below. Feedback was gathered in a short meeting rather than by email, which produced more detail.
39.4 Limits
One term of data; configuration state only, not attack resistance; scan may miss secrets that do not look random.
Configuration state shows that settings are correct, not that StudyHub would resist a determined attacker. Secrets that look like ordinary words, such as a simple password, would not trigger entropy rules. Samuel states these limits beside the results so readers do not overestimate what the report proves. He also notes that one term may include unusually active fixing because the project was new and visible. Whether the trend continues once the report becomes routine is something Tom's team will learn over the following terms.
39.5 Independent check
A cloud team member reruns the report from scratch using the guide and gets the same results.
Reproducing the results from the guide alone shows the report does not depend on Samuel's knowledge. It also tests the guide: one step needed clarifying, which was updated. This check matters most for handover, since Tom's team must run the report after Samuel leaves. The team member used a different laptop and fresh exports downloaded that day, so the check also showed the report did not rely on anything stored only on Samuel's machine. The results matched line for line, and the run took under ten minutes.
39.6 What next
Add Kubernetes role-binding analysis to the report; extend the scan to infrastructure code.
Role-binding analysis would bring the graph from the discrete structures chapter into the weekly report, catching paths like support-cy's automatically. Scanning infrastructure code would find secrets before they reach any image. Both are listed in the handover with estimated effort. Samuel also suggests checking whether the scan's allow list grows over time, since a growing list could signal that real secrets are being allowed through as placeholders. He adds a line to the weekly report showing the list's size, so the trend is visible to the owner without extra work.
Measuring user value
Beyond criteria, Samuel asks how the report is used. The cloud team says they read the summary every Monday and assign new findings in their planning meeting. Two developers say the report helped them find their own issues before reviews. Usage is the best sign the report will continue after the project. Samuel also checks how many findings were closed by owners without prompting: 22 of 32, which suggests the owner column works.
Survey
A short survey of the cloud team rates the report's clarity 4.2 of 5 and suggests adding links from each finding to its runbook.
Next owner
Tom names the report owner.
Worked example
Tom presents the evaluation to Westmere's IT leadership, who fund continuing the report and the student volunteering scheme. Samuel's limits section is quoted in the presentation, which Tom says increased leadership trust.
Synthetic case. Results are synthetic.
Errors and limits
Meeting criteria shows the project did what it set out to do; it does not show the criteria were the right ones.
Exercises
- EX115 — Findings trend. How did open findings change? Rubric: 3 = 41 to 9; 2 = partial; 1 = vague; 0 = none.
- EX116 — Median age. What was the median findings age? Rubric: 3 = 9 days; 2 = partial; 1 = vague; 0 = none.
- EX117 — Limits. Name two limits. Rubric: 3 = one term, config only, scan misses non-random secrets; 2 = one; 1 = vague; 0 = none.
Synthesis
Evaluate honestly against criteria and say what remains unknown.
Continue studying — chapter 39 of 40
← Chapter 38 (draft) · Course chapter index · Chapter 40 (draft) →