◇ OpenCyber Academy
Technologist · Chapter 30 · English

Chapter 30 — Capstone assessment

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 30 — Capstone assessment

Lesson map

Lesson 30 published. Map, example, EX088.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["30.1 The design review"]
  N2["30.2 Operational evidence"]
  N3["30.3 Assessment rubric"]
  N4["30.4 Common capstone failures"]
  N5["30.6 Looking back: the method"]
  N6["30.7 Continuing practice"]
  N7["30.8 Practical activity: self-assessment"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX088"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["30.1 The design review"]
  N2["30.2 Operational evidence"]
  N3["30.3 Assessment rubric"]
  N4["30.4 Common capstone failures"]
  N5["30.6 Looking back: the method"]
  N6["30.7 Continuing practice"]
  N7["30.8 Practical activity: self-assessment"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX088"]

[Learning outcomes and connection]
    |
    v
[30.1 The design review]
    |
    v
[30.2 Operational evidence]
    |
    v
[30.3 Assessment rubric]
    |
    v
[30.4 Common capstone failures]
    |
    v
[30.6 Looking back: the method]
    |
    v
[30.7 Continuing practice]
    |
    v
[30.8 Practical activity: self-assessment]
    |
    v
[Worked example] --> [Exercise EX088]

Learning outcomes and connection

After this chapter you should be able to:

The course's assessment plan has three parts: an integration project (Chapter 29), operational evidence (the evidence record and the results of tests over time) and a design review. This chapter explains how each is assessed and closes the book by looking back over the method it has taught.

30.1 The design review

A design review is a structured conversation in which reviewers test whether the service meets its requirements and preserves its key properties. Prepare: architecture with zones, conduits and management paths; the direction-of-trust property and how each change preserves it (Chapter 8); the top risk scenarios and the control set for each (Chapters 13 and 18); data-flow register (Chapter 12); accepted risks with owners. Reviewers will ask "what happens if…?" questions: if the relay is compromised; if a key leaks; if the carrier fails; if the automation's source of truth is wrong; if the person who built it leaves. Answer with the design and the evidence, and say "we have not tested that" when it is true — an honest gap scores better than a confident guess.

30.2 Operational evidence

Operational evidence is the record of the service running: health reports over weeks, alerts and their handling, control tests with results against claim rules (Chapter 23), restore tests, access reviews, drift reports, problem records, and blameless reviews with their changes. Assessors look for: evidence that is dated and sourced; counts that add up; claim rules stated before results; "not yet shown" handled honestly; and limits stated (synthetic data, lab equipment, short periods).

Team B's summary in this chapter is the model: counts, a digest, a claim rule and a stated gap.

30.3 Assessment rubric

Criterion 3 — strong 2 — adequate 1 — weak 0 — missing
End-to-end function All requirements of Chapter 29 ("What the capstone must demonstrate") demonstrated with raw evidence One requirement with thin evidence Function asserted, not shown Not working
Controls and monitoring Each top scenario has prevent/detect/recover controls tested against claim rules; self-monitoring present Gaps in one scenario or no self-monitoring Controls configured but untested None
Design properties Direction of trust and layer independence preserved and argued; management paths shown One property weakly argued Diagram only None
Operability and handoff Runbooks tested by others; health evidence; handover with open items Runbooks untested by others Documents only None
Evidence quality Dated, sourced, adds up, digested, limits stated One weakness Several weaknesses None
Honesty about limits Gaps and "not yet shown" stated plainly with plans Stated but without plans Gaps hidden —
Safety and ethics All practical work within authorization; no real systems touched — Minor lapse Unauthorized activity (fails the capstone)

30.4 Common capstone failures

The capstones that do worst are rarely the least technical. They are the ones that present configuration as evidence, hide a failed test, choose claim rules after the results, test only components, or describe a design that the evidence does not match. The capstones that do best are often modest in scope and exact in evidence.

Team A, "fully secured… all tests passed", shows the first two failures.

Team B, in contrast, reports Rule C's extra tests, FIM's 4 of 6 and an untested key compromise.

30.5 Worked example: two capstone summaries (synthetic)

Team A: "LANTERN is fully secured with HMAC, validation, monitoring and backups. All tests passed." Team B: "Four of four injected faults were detected in the final run (record digest aa40e3905d17…). Rule C met its claim rule only after further tests (12/12 after four more successes, lower bound 0.76); FIM detection is not yet shown (4/6, daily checks too slow — change raised). We found and fixed a replay-state defect during integration (Chapter 29). The historian restore met its 4-hour RTO in 9 of 10 drills; the failure was a runbook gap, fixed and re-tested. We have not tested key compromise or carrier failure at scale." Team B's summary scores higher on every criterion except perhaps confidence — and confidence is not a criterion.

30.6 Looking back: the method

The course has taught one method, applied thirty times: know what must stay true about the service; build it so that it stays true; check that it does, with evidence someone else can verify; notice quickly when it stops; respond without destroying evidence; and hand it on so that others can do the same. The vocabulary changes by sector — chlorine and pumps in water, drug libraries in a hospital, timetables at a depot — but the method transfers.

PS-09 traced the whole method once: silent, reconciled, alerted on, explained by its clock.

30.7 Continuing practice

Services and threats change; so do standards. Keep a short continuing-practice plan: re-read the main standards you rely on when they are revised (NIST CSF 2.0, SP 800-82, the CISA CPGs, the sector guidance for your jurisdiction); follow the advisories for the products in your inventory; take part in exercises; review one public incident report a month for failed assumptions (Chapter 5); and seek review of your own work. The next course in the study path, the Cybersecurity bachelor's course, broadens the foundations behind much of what this book applied.

30.8 Practical activity: self-assessment

Score your capstone against the assessment rubric above before your assessor does. For every criterion below 3, write what evidence would raise it and when you could produce it.

Use Team B's summary as the standard for the evidence each score needs. If Rule C met its claim rule in your record only after extra tests, say so and give the counts, as Team B did with 12 of 12 and lower bound 0.76. If FIM is not yet shown in your record, score that criterion honestly and name the change that would raise it, such as more frequent integrity checks on critical files, with a date for the retest. A self-assessment that differs from the assessor's by more than one point on any criterion usually means evidence was missing, not that the work was weak.

Exercises — chapter 30

EX088 — Answer the reviewers. Objective: defend a design honestly. Inputs: questions: (a) "What happens if the relay's HMAC key leaks?"; (b) "What if the person who built the automation leaves?"; (c) "What if BayTel is down for a day?" Constraints: answer from the design and evidence in this book; say where evidence is missing. Steps: answer each in 100–150 words. Expected evidence: three answers. Rubric: 3 = (a) per-station keys limit impact; detection via unexpected source or sequence anomalies is weak — admits gap; rotation runbook tested; revocation steps; (b) review, rotation of roles, runbooks, protected pipeline, handover — with evidence of shadowing; (c) stations buffer 24 h, silence alerts, manual readings trigger, backlog upload headroom — and states whether a day-long outage has been exercised; 2 = one answer weak; 1 = confident without evidence; 0 = none.

EX089 — Assess Team A and Team B. Objective: apply the rubric. Inputs: Chapter 30 ("Worked example: two capstone summaries") summaries. Constraints: score only what the summary shows. Steps: score both on each criterion and justify. Expected evidence: table. Rubric: 3 = Team A low on evidence quality and honesty (assertions only), unknown on others; Team B high on evidence quality and honesty, adequate-to-strong on controls with stated gaps; assessor notes what further evidence is needed for each; 2 = scores defensible, justification thin; 1 = scores by confidence of tone; 0 = none.

EX090 — Continuing-practice plan. Objective: plan to stay current. Inputs: your inventory's products and your jurisdiction's sector guidance. Constraints: one page; monthly, quarterly and yearly activities; named sources with versions. Steps: write the plan. Expected evidence: plan. Rubric: 3 = specific sources and versions (for example NIST CSF 2.0, SP 800-82 Rev. 3, CISA CPGs or NCSC CAF), advisory subscriptions for inventory products, monthly incident-report reading, quarterly exercise participation, yearly review of standards, peer review; realistic workload; 2 = generic sources; 1 = vague intent; 0 = none.

Chapter synthesis

The capstone is assessed through design review, operational evidence and the integration project. Reviewers reward designs that preserve their key properties, controls tested against claim rules stated in advance, evidence that is dated, sourced, consistent and protected, and honesty about what has not been shown. The method of the course — know what must stay true, build for it, check it, notice failure, respond with care, hand it on — transfers across sectors and must be kept current by continuing practice.

Continue studying — chapter 30 of 30

← Chapter 29 · Course chapter index · Next course via chapter index →

This is the final chapter of this course; the index links the next course in the full study path.

Worked answers

EX088 — Answer the reviewers. (a) Key leak: keys are per station, so a leaked key lets an attacker impersonate one station, not all; forged messages would pass MAC and validation if plausible; detection is weak — duplicate sequence numbers from two sources or values disagreeing with neighbouring stations might show it, but this has not been tested; response: revoke and rotate using the tested runbook, and treat that station's data as unreliable until rotated. (b) Builder leaves: desired state and automation are in a reviewed repository with at least two maintainers; runbooks were followed by the receiving team during shadowing; credentials are in a vault, not with the person; access is removed through the leaver process. (c) BayTel down for a day: stations buffer 24 h; silence alerts fire; Water switches to manual readings; on return, backlog upload has been tested for 4 h but not for 24 h — an honest gap, with a test planned.

EX089 — Assess Team A and Team B. Team A: end-to-end function 1 (asserted), controls 1 (configured, no test results), design properties — unknown, operability — unknown, evidence quality 0–1 (no dates, counts or digest), honesty 1 (no gaps stated). Team B: function 3 (four of four with digest), controls 2–3 (claim rules applied, FIM not yet shown with change raised), design properties — not shown in the summary, operability 2 (runbook gap found and fixed), evidence quality 3, honesty 3 (untested areas named). The assessor asks Team A for raw evidence and Team B for its design-review material before final scores.

EX090 — Continuing-practice plan. Monthly: read the advisories for products in the inventory (vendor notices; national CERT feeds; CISA KEV) and one public incident report, noting failed assumptions; review health-report trends. Quarterly: take part in one exercise or control test; review access-review and drift findings with a peer. Yearly: re-read the versions in use of NIST CSF 2.0, SP 800-82 Rev. 3 and either the CISA CPGs or the NCSC CAF, and the sector guidance for the jurisdiction; check for revisions; update the control cards affected; seek an external review of one piece of your own work. Workload: about four hours a month.