Chapter 46 — Engineering capstone II
Lesson map
Lesson 46 published. Map, example, EX136.
flowchart TD
N0["Learning outcomes"]
N1["46.1 Build"]
N2["46.2 Pilot"]
N3["46.3 Rollout"]
N4["46.4 Verification"]
N5["46.5 Validation"]
N6["46.6 Remaining gaps"]
N7["46.7 Report"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX136"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["46.1 Build"]
N2["46.2 Pilot"]
N3["46.3 Rollout"]
N4["46.4 Verification"]
N5["46.5 Validation"]
N6["46.6 Remaining gaps"]
N7["46.7 Report"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX136"]
[Learning outcomes]
|
v
[46.1 Build]
|
v
[46.2 Pilot]
|
v
[46.3 Rollout]
|
v
[46.4 Verification]
|
v
[46.5 Validation]
|
v
[46.6 Remaining gaps]
|
v
[46.7 Report]
|
v
[Worked example] --> [Exercise EX136]
Learning outcomes
You will (1) build and roll out the capstone system; (2) verify it against requirements; (3) validate it with an authorised re-test; (4) document results and remaining gaps.
46.1 Build
The build follows the reviewed design: firewall logging changes, parser, enrichment link to the asset inventory, rule covering two protocols, and the playbook.
Following the reviewed design closely matters because the review's findings were resolved against that version. Any deviation during build goes back to the reviewer for a quick check. Each component is built and tested in the test environment first, in dependency order: logging before parsing, parsing before enrichment, enrichment before the rule, and the playbook last once alerts can be produced.
46.2 Pilot
Three depots pilot for two weeks. One false alert per week arises from a depot IT technician using a non-approved laptop; the technician's device is registered properly.
The false alert is a useful one: a technician using an unregistered laptop on the depot network is exactly the kind of device the approved list should know about. Registering it properly fixes the alert and improves the inventory. One false alert per week is well inside the target of five, which supports moving on to full rollout.
46.3 Rollout
Remaining depots follow in groups, with CPU monitored on each firewall; all stay below 2%.
Watching firewall CPU guards against the risk that extra logging slows depot networks and disrupts dispatch. Below 2% shows the change adds little load. Rolling out in groups means any problem affects only a few depots before it is caught, and the pilot's lessons are applied to each new group. Each group's figures are posted on the rollout dashboard, and depot managers receive a short note confirming their site has changed and whom to call if something looks wrong.
46.4 Verification
Unit and pipeline tests pass. Timed synthetic logons measure 9.8 to 13.1 minutes end to end. Completeness reconciliation shows 99.94%.
The timed results fall well inside the fifteen-minute requirement, even at the slowest. Completeness above 99.9% meets the target for priority sources. Passing unit and pipeline tests shows the logic and data flow behave as specified. Together these verify that the system was built as designed. Amara records the date, rule version and tester beside each result, so the evidence can be matched to the exact configuration that produced it, even after later changes.
46.5 Validation
The purple team re-runs the two techniques that previously left no trace; both now alert, and analysts follow the playbook correctly.
Verification showed the system matched its specification; validation shows it now catches the realistic behaviour that was missed before. Analysts following the playbook correctly shows the human side works too. Closing the two gaps found earlier is the central evidence that the capstone achieved its purpose. The purple team's report is filed in the dossier with the earlier report, so the before-and-after comparison is easy to make.
46.6 Remaining gaps
Logons using the domain controllers' web management interface are not covered; they are logged as a follow-up item.
Reporting what is still uncovered is part of honest engineering. The web management interface is a different path that the new rule does not watch. Logging it as a follow-up means it is tracked with an owner rather than forgotten, and readers of the report know the limits of what was delivered. Amara estimates the effort needed to cover that path and hands the item to Paolo's backlog with her notes.
46.7 Report
The capstone report presents requirements with pass or fail, evidence, and gaps, with all data synthetic or anonymised.
Presenting requirements with a clear pass or fail lets readers see results at a glance. Evidence references let them check each one. Gaps show what still needs work. Keeping all data synthetic or anonymised means the report can be shared with assessors without exposing Lindqvist's systems or customers. Paolo reads the draft before submission and checks that each pass claim matches its evidence, sending back two claims whose wording went further than the data.
Handover to operations
When the capstone ends, the rule, parser and playbook pass to the detection engineering team. Amara walks the on-call analysts through the playbook, adds the rule to the quarterly review list and records the owner. A system no one owns after the student leaves would decay quickly.
Lessons from the build
The largest delay came from waiting for change windows at depots, not from engineering. Amara notes for future projects that scheduling with operations should start in the design phase.
Worked example
Amara's results table: latency requirement met (max 13.1 min); false alerts 1, 0, 0, 1 per week (≤5 met); completeness 99.94% (met); CPU ≤ 1.7% (met); validation 2/2 techniques alerted. Gap: web management interface — open, owner assigned.
Synthetic case. Results are invented.
Errors and limits
Four weeks of operation is short; long-term false-alert rates may differ.
Exercises
- EX136 — Pilot false alert. What caused the pilot false alert, and how was it resolved? Rubric: 3 = technician's unregistered laptop; registered properly; 2 = partial; 1 = suppressed; 0 = none.
- EX137 — Requirements met. Summarise verification results against requirements. Rubric: 3 = latency ≤13.1, false ≤1/week, completeness 99.94%, CPU ≤1.7%; 2 = three; 1 = one; 0 = none.
- EX138 — Remaining gap. What gap remains, and how is it handled? Rubric: 3 = web management interface logons; follow-up item with owner; 2 = partial; 1 = vague; 0 = none.
Synthesis
Build, verify and validate against each requirement, and report remaining gaps openly.
Continue studying — chapter 46 of 50
← Chapter 45 (draft) · Course chapter index · Chapter 47 (draft) →