◇ OpenCyber Academy
Engineering · Chapter 46 · English

Chapter 46 — Engineering capstone II

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 46 — Engineering capstone II

Lesson map

Lesson 46 published. Map, example, EX136.

flowchart TD
  N0["Learning outcomes"]
  N1["46.1 Completing the build"]
  N2["46.2 Closing conditions"]
  N3["46.3 Stakeholder validation"]
  N4["46.4 Handover"]
  N5["46.5 Measured results"]
  N6["Practice note — known issues"]
  N7["Post-acceptance monitoring"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX136"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["46.1 Completing the build"]
  N2["46.2 Closing conditions"]
  N3["46.3 Stakeholder validation"]
  N4["46.4 Handover"]
  N5["46.5 Measured results"]
  N6["Practice note — known issues"]
  N7["Post-acceptance monitoring"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX136"]

[Learning outcomes]
    |
    v
[46.1 Completing the build]
    |
    v
[46.2 Closing conditions]
    |
    v
[46.3 Stakeholder validation]
    |
    v
[46.4 Handover]
    |
    v
[46.5 Measured results]
    |
    v
[Practice note — known issues]
    |
    v
[Post-acceptance monitoring]
    |
    v
[Worked example] --> [Exercise EX136]

Learning outcomes

You will (1) complete the capstone build to the charter; (2) close review conditions; (3) validate with stakeholders on synthetic scenarios; (4) prepare handover to operations.

46.1 Completing the build

The vendor-watch service ingests synthetic vendor advisories and partner reports, tags vendors and products, scores relevance against BEN's synthetic OT asset list, and opens OT tickets with recommendations, TLP marking and the two-person rule.

Each step was built from the thin slice the team finished in the first semester. Relevance scoring compares the vendor and product names in an advisory with the synthetic asset list, and the score decides whether a ticket opens. Nothing in the service can change an OT system; its output is always a recommendation that an engineer reads and decides on.

46.2 Closing conditions

Each must-fix finding is closed with evidence: a test, a screenshot of the ticket template, a code reference. The review chair confirms closure.

For the two-person rule, the evidence was a test that tries to close a ticket with one approval and checks that the system refuses. For the reminder in each recommendation, the evidence was the ticket template itself and a sample of generated tickets. The chair read each item and recorded her confirmation in the review file. No condition was closed by a statement alone, however confident the team felt about the fix.

46.3 Stakeholder validation

OT engineers run five scenarios. In four, recommendations are relevant and timely; in one, a vendor name variant was missed. The team adds the variant to the vendor dictionary and a rule to flag unknown names for analyst review.

The missed variant was a shortened form of a product name used in one advisory. A dictionary alone would always miss some new form, which is why the rule for unknown names matters more than the single addition. Analysts now review those flagged items, and each confirmed variant goes into the dictionary. The team reran all five scenarios after the change, and all passed.

46.4 Handover

Runbooks, on-call notes, configuration guide, known issues and a training session for the OT team. Operations signs acceptance.

The training session used the same synthetic advisories as the scenarios, so OT engineers saw the tickets they would receive in real use. Kofi's team walked through each known issue, including the dependence on the vendor dictionary, and explained the workaround for each. Operations signed acceptance only after a member of their own team followed the runbook to restart the service without help. The signed form is part of the evidence dossier.

46.5 Measured results

On the 200-item synthetic test set: 96% of relevant items alerted within 15 minutes; 7% of alerts irrelevant. Both meet the charter's objectives.

Both figures come from the same defined test set the charter named, so the comparison is fair. Kofi's report gives the counts behind the percentages and notes that a set of two hundred items cannot show how the service behaves on rare kinds of advisory. The service met its objectives on this set; whether it does so on real advisories is what the first months of monitoring will show.

Practice note — known issues

Listing known issues honestly at handover builds trust; operations will find them anyway.

Post-acceptance monitoring

For the first month after acceptance the team reviews every OT ticket the service creates with the OT engineers, comparing its relevance judgments with theirs. Disagreements feed the vendor dictionary and scoring rules, and the agreement rate is reported to the supervisor as an early operational measure.

Worked example

The final demonstration shows an advisory naming "RelayCo" (synthetic) flowing to an OT ticket in 6 minutes, with ATT&CK techniques from the advisory (for example T1190 Exploit Public-Facing Application) listed for context and the two-person approval enforced.

Synthetic case. RelayCo and results are invented.

Errors and limits

Synthetic test sets may not reflect real advisory diversity; plan to review performance on real public advisories after go-live. Acceptance is not the end of maintenance.

Exercises

Synthesis

Build to the charter, close conditions with evidence, validate with users, hand over honestly and measure against objectives.

Continue studying — chapter 46 of 50

← Chapter 45 (draft) · Course chapter index · Chapter 47 (draft) →