OpenCyber Academy
Bachelor's · Chapter 39 · English
Course code: OCA-111Chapter 39: OCA-111-C039

Chapter 39 — Capstone evaluation

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 39 published. Map, example, EX115.

flowchart TD
  N0["Learning outcomes"]
  N1["39.1 Plan"]
  N2["39.2 Independent"]
  N3["39.3 Side effects"]
  N4["39.4 Residual risk"]
  N5["39.5 Report"]
  N6["39.6 Recommend"]
  N7["39.7 Independence statement"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX115"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["39.1 Plan"]
  N2["39.2 Independent"]
  N3["39.3 Side effects"]
  N4["39.4 Residual risk"]
  N5["39.5 Report"]
  N6["39.6 Recommend"]
  N7["39.7 Independence statement"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX115"]

[Learning outcomes]
    |
    v
[39.1 Plan]
    |
    v
[39.2 Independent]
    |
    v
[39.3 Side effects]
    |
    v
[39.4 Residual risk]
    |
    v
[39.5 Report]
    |
    v
[39.6 Recommend]
    |
    v
[39.7 Independence statement]
    |
    v
[Worked example] --> [Exercise EX115]

Learning outcomes

You will (1) evaluate against pre-registered metrics; (2) run independent tests; (3) state residual risk.

39.1 Plan

Precision/recall on planted drifts; fail-closed cases.

The evaluation plan locks precision and recall on planted drifts plus fail-closed cases before any run begins. Sam freezes metric definitions in a short pre-registration note the advisor signs. Planted adds, removes and type conflicts provide ground truth the script can score. Fail-closed cases are scored as pass or fail, never as soft partial credit. Plans that invent metrics after seeing scores fail the course standard.

Sam plants twenty drifts across the inventory pair (eight adds, seven removes and five type conflicts) so each class has enough cases for an interval to mean something.

39.2 Independent

Another student runs the evaluation script.

Another student runs the evaluation script while Sam stays away from the keyboard. Independence is a social protocol as much as a technical one. The runner signs a one-line statement that Sam did not edit the run logs. Statements sit in the appendix beside the raw outputs. Coached runs are treated as academic misconduct, not as helpful collaboration.

The runner receives only the README, the frozen script and the fixture bundle; if they get stuck, they file a question in writing, and the question itself becomes evaluation data about the documentation.

39.3 Side effects

Usability timing; false alarm burden.

Side-effect measures include usability timing and false-alarm burden for operators. A precise tool that floods operators with noise will not be adopted. Sam times first-time conflict triage on a fixed inventory pair. False alarms are counted and discussed even when precision looks flattering. Side effects belong in the report beside the headline scores.

Sam's first measurement showed operators spending most of their time on type conflicts that were harmless naming variants. He reported that cost honestly instead of tuning it away before the evaluation closed, and the finding went into the v2 list.

39.4 Residual risk

Residual risk statements say clearly that the tool helps inventory hygiene and does not replace ROE discipline. Sam refuses marketing language that implies authorised testing can be automated away. Readers who skip this paragraph are the ones most likely to misuse the tool. The statement is required in both the report abstract and the final slide. Hygiene aids remain subordinate to human authorisation boundaries.

A concrete example helps: the tool can say a host is missing from the documented scope, but it cannot say whether testing that host is authorised. That judgement stays with the ROE owner.

39.5 Report

The report compares results to the pre-registration and forbids post hoc metric swaps. Exploratory analyses are allowed only in a clearly labeled appendix. Sam's template highlights any metric that was not pre-registered. Victory-lap swaps are a fail even if the new metric looks kinder. Honesty about misses is part of the graded science.

Where a planted drift class scored poorly, the report shows the misses by fixture name so readers can inspect them. Sam's lowest result, on type conflicts, appears in the abstract rather than on the last page.

39.6 Recommend

Adopt in staging first.

The recommendation is staging adoption first with named owners and a thirty-day checklist. Tools without adoption plans are hobbies, however clever. Sam writes what staging success looks like before anyone talks about production. Production promotion waits on evidence from solo operator runs. Recommendations that skip staging are rejected by the advisor.

In Sam's plan, staging success means two solo operator runs with no author help, no fail-closed regressions, and false alarms below the level operators said they could tolerate in the timing study. Only then does a production discussion start, with the service owner chairing it.

39.7 Independence statement

Independent evaluators sign a one-line statement that Sam did not edit their run logs. The statement sits in the appendix. Trustworthy evaluation is a social protocol as much as a script.

Scripts alone cannot prove independence when authors linger nearby. The signed line makes the social rule visible to examiners. Sam keeps distance during runs even when curiosity itches. Missing statements block the evaluation chapter from closing. Trustworthy science is partly paperwork done without resentment.

The wording is fixed so that runners cannot soften it into "mostly unedited".

39.8 Staging adoption plan

Evaluation reports include a thirty-day staging plan with owners. Tools without adoption plans are hobbies.

The thirty-day plan names dates, owners and evidence expected at each gate. Sam rejects plans that only say monitor in staging. Checklists become tickets so work cannot hide in email bullets. Hobby energy is welcome in labs and unwelcome in adoption claims. Staffed plans are how tools leave the thesis and enter operations.

Sam's plan has three gates: a first solo run by day ten, a second by day twenty, and a review of break reports and false-alarm counts at day thirty. Each gate names the evidence file that proves it passed.

39.9 Adoption owners

Staging adoption names a service owner and a platform owner. One name is insufficient.

Service owners feel the operational pain; platform owners feel the tooling pain. One name usually means the student still owns everything unofficially. Sam requires both roles before the staging plan can be approved. Split ownership prevents silent abandonment when the term ends. Insufficient naming is treated as an incomplete recommendation.

In practice the service owner decides whether a conflict list is good enough to act on, while the platform owner fixes parser and packaging issues. When those two jobs share one name, operational complaints tend to go unanswered because the same person is also busy fixing code.

39.10 Mira and Sam on adoption

Adoption plan names Mira for service ownership and Sam for platform support. Thirty-day checklist includes two operator solo runs. Without Mira's name the plan would be fiction.

Mira's name makes the plan real because she can refuse work that is not ready. Two solo runs catch documentation gaps unit tests miss. Sam's platform role stays supportive rather than possessive. Fiction plans with only student names are a known failure mode. The scene is used in advisor training as a positive example.

39.11 Solo run finds DNS gap

Solo run two failed because DNS differed in the operator's profile. README gained a DNS preamble. Solo runs find doc bugs tools miss.

Profile differences are boring until they break a careful operator. Sam thanked the failure in the retro instead of blaming the operator. DNS preamble text is now part of the golden README examples. Solo runs are documentation tests wearing operational clothing. Tools can pass unit tests and still fail humans on day one.

The operator's resolver returned short names where the fixture expected fully qualified ones.

39.12 Checklist items as tickets

Thirty-day checklist items are tickets, not bullets in an email.

Tickets carry owners, due dates and evidence attachments. Email bullets evaporate when inboxes rotate. Sam's adoption template opens tickets automatically from the checklist. Closed tickets without evidence are reopened by policy. Ticketing is how thirty days become accountable instead of aspirational.

Sam's checklist produces six tickets: two solo runs, the false-alarm review, a deny-message review, the README update and the day-thirty retro. Each has an owner from the adoption plan and a due date inside the thirty days, so slippage is visible on the board before the window closes.

39.13 Evidence to auto-close

Tickets for checklist items auto-close only when solo-run evidence is attached.

Auto-close without evidence recreates paper completion theatre. Sam wires the rule so convenience cannot delete the proof requirement. Operators learn to attach outputs while the run is fresh. Missing attachments keep the ticket loud on purpose. Evidence-gated closure is adoption hygiene.

Acceptable evidence is specific: the conflict output file, the operator's command line and the tool version hash. A screenshot alone does not count, because it cannot show which build ran. When evidence is missing, the ticket stays open and pings its owner each morning.

39.14 Rewrite ignored README parts

Adoption retros at day 30 ask operators what they ignored in the README. Ignored sections are deleted or rewritten, not defended.

Defending unread paragraphs wastes the only attention operators offered. Sam would rather shrink docs than lecture people for skimming. Ignored means the writing failed the environment it entered. Retros that rewrite are kinder than retros that blame. Attention is the scarce resource adoption must budget.

In the first retro, operators admitted skipping the long architecture overview entirely. Sam cut it to three sentences and moved the deny-path example to the top, since that was the part people actually needed during their first run.

39.15 Intentional break reports

Day-ten adoption check asks operators to break the tool on purpose and file the brokenness. Sam would rather hear early pain than polite silence. Two intentional break reports in the first cohort improved deny messages more than a month of speculative polish would have.

Polite silence is an adoption anti-pattern. Inviting breaks legitimises early bad news. Deny-message improvements from real confusion beat imagined UX polish. Sam tracks break reports as valued input, not as complaints. Early pain is cheaper than quiet abandonment.

39.16 BREAK-ADOPT ticket type

Intentional break reports have their own ticket type: BREAK-ADOPT.

A distinct type keeps adoption pain searchable later. Sam reviews BREAK-ADOPT trends each cohort for doc and deny gaps. Mixing breaks into generic bugs hides the adoption signal. Operators use the type without shame because it was requested. Searchable pain is how programmes learn across years.

Each report records what the operator tried, what the tool did and what they expected instead. The first cohort's reports found that an empty inventory file produced a vague message, which became a specific deny reason in the next release.

Worked example (synthetic TRB)

Independent run: precision 0.91 recall 0.88 on planted set; 3/3 corrupt files denied; Sam recommends staging adoption.

Synthetic case — Riverstone Transit Bank (TRB). The bank, Sam Ito, TrackScope/FindingsDesk/PulseMap and all figures are invented for teaching.

Errors and limits

This book grants no authority to test real banks or payment systems. Labs use documentation-range addresses and synthetic data only; exploit code is never provided.

Exercises

Synthesis

Evaluation is independent, pre-registered and humble about residual risk.

Continue studying — chapter 39 of 40

← Chapter 38 (draft) · Course chapter index · Chapter 40 (draft) →

Worked answers

EX115. Another student or independent tester.

EX116. The tool does not replace ROE discipline or authorization checks.

EX117. No. Metrics stay as pre-registered.