Chapter 30 — Capstone assessment
Lesson map
Lesson 30 published. Map, example, EX088.
flowchart TD
N0["Learning outcomes"]
N1["30.1 Assessment plan"]
N2["30.2 Client validation"]
N3["30.3 Results"]
N4["30.4 Live event"]
N5["30.5 Reflection"]
N6["30.6 Presentation"]
N7["30.7 Handover"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX088"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["30.1 Assessment plan"]
N2["30.2 Client validation"]
N3["30.3 Results"]
N4["30.4 Live event"]
N5["30.5 Reflection"]
N6["30.6 Presentation"]
N7["30.7 Handover"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX088"]
[Learning outcomes]
|
v
[30.1 Assessment plan]
|
v
[30.2 Client validation]
|
v
[30.3 Results]
|
v
[30.4 Live event]
|
v
[30.5 Reflection]
|
v
[30.6 Presentation]
|
v
[30.7 Handover]
|
v
[Worked example] --> [Exercise EX088]
Learning outcomes
You will (1) assess the capstone against requirements with evidence; (2) validate on client systems with authorisation; (3) reflect on what worked and what did not; (4) present results to the SOC and assessors.
30.1 Assessment plan
Each requirement is checked with lab results, authorised client tests and one month of live operation across twelve healthcare clients.
Each kind of evidence in the plan covers something that the others cannot. Lab results show the components work on known events, client tests show that each client's systems produce and deliver the events, and a month of live operation shows how often the package pages without a real attack. Twelve healthcare clients are the population the package was built for.
30.2 Client validation
With agreement from each client, Copperleaf runs the account test (chapter 27) and a stopped-agent test during maintenance windows. Rename spikes are tested only in the lab, since generating them on client systems would disrupt work.
The account test and stopped-agent test are both harmless and reversible, so they can run on live systems with the client's agreement. A rename spike cannot be produced without changing hundreds of real files, which is why that test stays in the lab, the trade-off Mei later explains to an assessor in the worked example.
30.3 Results
Sequence alerts arrived in 1–4 minutes at all twelve clients. Heartbeat pages arrived in 11–14 minutes. In live operation, false pages averaged 0.4 per client per month. Rename isolation is verified in the lab only.
Mei sets each of these figures against the requirement it is meant to meet. One to four minutes is within the five-minute target for sequence alerts, 11 to 14 minutes is within the 15-minute target for heartbeat pages, and 0.4 false pages per client per month is within the limit of one. Only rename isolation has no live figure, and Mei says so.
30.4 Live event
One real event occurred: a clinic PC's new admin account at night, caught by the sequence rule and contained before any file activity. The account belonged to an attacker using a phished contractor password.
This event matches the supplier scenario from chapter 13: an attacker using the contractor's stolen password. The sequence rule's contractor exception did not hide it, because the exception applies only inside agreed support windows from the contractor's VPN addresses, and this logon was at night. Containment before any file activity is what the sequence rule was built to achieve.
30.5 Reflection
What worked: reusing tested components, staged deployment. What did not: the first heartbeat interval; one client's maintenance window was undocumented, causing a false page.
Both failures were found by the evaluation process rather than by clients. The five-minute heartbeat interval missed the 15-minute target by one minute in the lab and was shortened to two minutes, and the false page from an undocumented maintenance window points to recording every client's maintenance windows before go-live. Mei records both points in her report, so the future owner knows the heartbeat interval was changed and why.
30.6 Presentation
Mei presents to Copperleaf's team and her assessors: requirements, evidence, the live event, limits and maintenance plan.
Mei's presentation follows the order of the assessment criteria, so assessors can match each part to what they grade. The live event shows the package working on a real attack, and the limits, such as rename isolation verified only in the lab, are stated by Mei before anyone has to ask. The presentation uses the same requirement table as the report, so figures quoted aloud can be checked against the written evidence.
30.7 Handover
The package passes to a named engineer with a review date.
The handover repeats the ownership pattern from chapter 26: a named engineer, a review date and the runbooks with their change history. When Mei's placement ends, the package keeps working because its owner has the requirements, test records and version 1.0 tag needed to maintain it and judge future changes against. The engineer receives the package in version control with its 1.0 tag, the validation records from client tests and the month of live results. The review date is set so the first review falls after another period of live operation, when new figures can be compared with the assessed month.
Assessor criteria
Assessors grade clarity of requirements, quality of evidence, honesty about limits and the maintenance plan.
Next steps
After assessment, the package enters normal review cycles with the named engineer.
Worked example
An assessor asks why rename isolation was not tested on client systems. Mei explains that creating hundreds of renamed files on a live clinic share risks disrupting patient care; lab evidence plus the guard rail is the proportionate choice, and the limitation is stated in her report.
Synthetic case. The assessment is invented.
Errors and limits
One month and twelve clients give an early picture; continue measuring.
Exercises
- EX088 — Results summary. Summarise the capstone results. Rubric: 3 = sequence 1–4 min; heartbeat 11–14 min; 0.4 false pages; rename lab only; 2 = partial; 1 = one; 0 = none.
- EX089 — Live event. Describe the live event caught. Rubric: 3 = night admin account by phished contractor password; contained before files; 2 = partial; 1 = vague; 0 = none.
- EX090 — Lab-only test. Why was rename isolation tested only in the lab? Rubric: 3 = live test could disrupt patient care; proportionate; stated limit; 2 = partial; 1 = forgot; 0 = none.
Synthesis
Assess against requirements with proportionate evidence, report limits and hand over for ongoing ownership.
Continue studying — chapter 30 of 30
← Chapter 29 (draft) · Course chapter index · Worked results ↓ · Next course via chapter index →