Chapter 38 — Capstone implementation
Lesson map
Lesson 38 published. Map, example, EX112.
flowchart TD
N0["Learning outcomes"]
N1["38.1 Build"]
N2["38.2 Fail closed"]
N3["38.3 Tests"]
N4["38.4 Docs"]
N5["38.5 Freeze"]
N6["38.6 Handover"]
N7["38.7 Tricky golden fixtures"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX112"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["38.1 Build"]
N2["38.2 Fail closed"]
N3["38.3 Tests"]
N4["38.4 Docs"]
N5["38.5 Freeze"]
N6["38.6 Handover"]
N7["38.7 Tricky golden fixtures"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX112"]
[Learning outcomes]
|
v
[38.1 Build]
|
v
[38.2 Fail closed]
|
v
[38.3 Tests]
|
v
[38.4 Docs]
|
v
[38.5 Freeze]
|
v
[38.6 Handover]
|
v
[38.7 Tricky golden fixtures]
|
v
[Worked example] --> [Exercise EX112]
Learning outcomes
You will (1) build the scope-diff tool; (2) test fail-closed paths; (3) document for operators.
38.1 Build
Diff two synthetic inventories; flag only-in-docs / only-on-network / type conflicts.
The implementation diffs two synthetic inventories and flags hosts only in documents, only on the network, or present in both with conflicting types. Sam keeps the first vertical slice tiny: load, compare, print conflicts, exit non-zero on corruption. Pretty dashboards wait until the core comparison is tested. Inventory field lists are pinned in a config file rather than hard-coded magic. Operators should understand a conflict line without reading the source. A clear conflict list is the product; everything else is optional clothing. Shipping the boring diff first prevents a museum of unfinished interface work.
38.2 Fail closed
Corrupt YAML → deny with reason.
Corrupt YAML must deny with a reason string a human can act on. Returning an empty allow list on parse failure is treated as a critical bug. Sam's property tests flip bits until deny triggers on every mutation class. Reasons name the file and the failure class without dumping secrets. Fail-closed is the first demo at every checkpoint, not an afterthought. Parsers that helpfully skip bad documents are unsafe in this context. Denial with a reason is success when the input cannot be trusted. Deny paths earn trust when reasons stay actionable without leaking secrets.
38.3 Tests
Golden fixtures; no network.
Tests use golden fixtures and never touch the network. Fixtures include happy inventories, near-matches, truncated files and wrong encodings. Sam names corrupt fixtures with a corrupt- prefix so CI properties can find them. Network calls in the test tree fail the build on sight. Golden outputs are reviewed like findings: small, explicit and hashed. Offline tests keep evaluation honest on locked-down lab machines. Fixtures are the curriculum for the tool's future maintainers. Fixture work earns trust when hashes make quiet edits obvious to reviewers.
38.4 Docs
Operator README with examples.
The operator README shows copy-pasteable examples for the common conflict cases. Sam writes the README as if the reader has never seen his shell aliases. Examples include the deny path so operators practice failure, not only success. Screenshots are optional; commands and expected output are required. A README that only works beside Sam's laptop fails the handover exam. Docs are updated in the same pull request as behaviour changes. If an operator needs tribal knowledge, the README is unfinished. Documentation earns trust when strangers succeed without tribal aliases.
38.5 Freeze
Feature freeze before evaluation week.
Freezing the build also keeps the planted drifts and fail-closed cases measuring the same code the advisor signed off in the pre-registration note. Sam parks ideas on a v2 list rather than merging chrome during measurement. Freeze breaks are allowed only for fail-closed defects. The advisor enforces freeze with the same seriousness as a production change window. Students who keep coding features into evaluation week void their pre-registration. Calm measurement needs a still codebase. Freeze is how evaluation stays about the tool that was promised. Freeze discipline earns trust when measured builds match promised builds.
38.6 Handover
Shadow an operator running it alone.
Handover means shadowing an operator who runs the tool alone on a fresh account. Sam may answer questions but may not type. If the operator needs shell history full of Sam aliases, the docs failed. The session is timed and noted like the leave-behind test from field practice. Successful handover is a graded outcome, not a courtesy demo. Clean accounts tell the truth about documentation quality. Alone and successful is the only handover result that counts. Handover work earns trust when fresh accounts finish without author hands.
38.7 Tricky golden fixtures
Golden fixtures include a deliberately corrupt scope and a near-match inventory designed to trick fuzzy matching. Sam's tool must not "helpfully" merge near-matches without an explicit flag.
Helpful merges are how distinct hosts become one confused record. The near-match fixture exists to punish default fuzziness. Sam's tests fail if conflicts disappear without --ack-fuzzy. Corrupt scope fixtures must deny even when they look almost valid. Fixtures that try to trick you are acts of friendship to future operators. Default suspicion on near-matches is a safety feature, not a UX bug. Near-match fixtures earn trust when defaults refuse silent collapse.
38.8 Fuzzy match flag
Near-match inventory names require --ack-fuzzy to merge. Default is to list them separately as conflicts.
The flag forces an intentional choice before inventories collapse. Default conflict listing keeps staging and production twins from merging silently. Sam logs flag use with username for later audit questions. UI shortcuts that skip the flag are forbidden in v1. Operators who need merges often should fix naming hygiene, not weaken defaults. Friction before merge has already saved a production host from a staging note. Fuzzy flags earn trust when intentional merges leave an audit trail.
38.9 Corrupt fixture names
Fixtures named corrupt-*.yaml must deny. A CI property test generates random corruption.
Naming conventions let property tests discover fixtures without a fragile manifest. Random corruption grows each term as parsers harden. Sam requires every new mutation to have a short intent name in the test output. A fixture that allows under the corrupt- prefix fails CI hard. Convention plus properties beat hand-maintained allow lists of bad files. Discoverable unsafe fixtures keep the safety net from rotting. Corrupt prefixes earn trust when properties keep discovering new danger.
38.10 Bit-flip property test
Random corruption property test flips bits in YAML until deny triggers. Sam watches it find a parser path that returned empty allow. Property tests are rude and helpful.
The empty allow path would have looked like a clean empty scope in production. Rudeness in tests is welcome when the alternative is polite blindness. Sam added a regression fixture named after the mutation that found it. Watching the test work teaches more than a slide about fail-closed. Helpers that return empty on error are guilty until proven deny. Rude tests earn their keep the first time they embarrass a parser.
38.11 Truncation and encoding fixtures
Corrupt fixtures include truncated files and wrong encodings. Deny must trigger on all. Sam's property test grows a new mutation each term.
Truncation mid-document and surprising encodings are common in real exports. Deny must not depend on a single corruption style. Each term's new mutation is recorded with a one-line intent for CI readability. Sam refuses to delete old mutations when new ones arrive. Growing the zoo is how confidence grows. Encoding traps belong beside bit flips in any serious parser suite. Encoding suites earn trust when deny does not depend on one mutation style.
38.12 Named mutations
Each new mutation in the property test gets a name and a one-line intent.
Names like trunc-mid-yaml make failures readable in CI logs. Intent lines stop future students from deleting a mutation they do not understand. Sam reviews names for clarity the same way he reviews finding titles. Unnamed mutations are rejected in code review. Readable failures shorten the loop from red CI to fixed parser. If the failure line needs a translator, the mutation is under-documented. Named mutations earn trust when CI failures explain themselves in plain words.
38.13 Readable mutation names
Mutation names look like "trunc-mid-yaml" so failures read clearly in CI.
Opaque hashes as mutation ids waste minutes during evaluation week. Sam's style guide for tests bans cute names that hide intent. Clear names also help advisors grade whether the suite covers real risks. Renaming for clarity is an encouraged refactor. CI is a communication surface, not only a gate. People read failure lines under stress; write them for that reader. Readable names earn trust when advisors can grade coverage from log lines.
38.14 Fresh account operator day
Operator day uses a fresh account with no Sam aliases in shell history. If the operator needs history, docs failed. Clean accounts are honest tests.
Shared tribal shells lie about readiness. Fresh accounts strip the comfort of incomplete docs. Sam watches without typing and takes notes for README repairs. Aliases that sneak back via dotfiles are in scope for the honesty check. The awkwardness is the point of the exam. Honest handover tests are kinder than surprising a night operator later. Fresh accounts earn trust when honesty about docs survives social awkwardness.
38.15 Typed merge acknowledgement
Fuzzy-match acknowledgements require typing the phrase "I accept merge risk" exactly. Copy-paste from a wiki is fine; clicking through is not. Sam wants a moment of friction before inventories collapse distinct hosts into one. That friction has prevented at least one production host from inheriting a staging-only note.
Click-through dialogs train blindness; typed phrases create a pause. Wiki copy-paste still forces the eyes across the words. Audit logs store who typed the phrase and when. Removing the phrase for convenience is a product defect. Designed friction is an engineering control when merges are dangerous.
38.16 Acknowledgement audit log
Typed acknowledgement phrase is recorded in the audit log with username.
Logs support later questions when a bad merge is discovered. Usernames come from the authenticated operator context, not from a free-text box. Sam's runbooks tell incident responders where to find the phrase events. Log retention follows the same vault rules as other engagement evidence. Without the log, the typed phrase is only theatre. Evidence of friction matters as much as the friction itself. Responders should reach the field names without asking Sam where he hid them. Acknowledgement logs earn trust when responders can find fields without asking.
Worked example (synthetic TRB)
Sam's tool on synthetic data matches lab_inv-style statuses; corrupt file exits non-zero.
Synthetic case — Riverstone Transit Bank (TRB). The bank, Sam Ito, TrackScope/FindingsDesk/PulseMap and all figures are invented for teaching.
Errors and limits
This book grants no authority to test real banks or payment systems. Labs use documentation-range addresses and synthetic data only; exploit code is never provided.
Exercises
- EX112 — Statuses. Name three diff statuses the tool prints. Rubric: 3 = only-in-docs, only-on-network, type conflict/match; 2 = two; 1 = one; 0 = none.
- EX113 — Corrupt file. What should happen on corrupt YAML? Rubric: 3 = deny/fail closed with reason, non-zero exit; 2 = partial; 1 = empty success; 0 = none.
- EX114 — Network. May the tool open network sockets? Rubric: 3 = no; it must run offline on local files; 2 = no, without explanation; 1 = yes; 0 = none.
Synthesis
Implementation is offline, tested, fail-closed and operator-documented.
Continue studying — chapter 38 of 40
← Chapter 37 (draft) · Course chapter index · Chapter 39 (draft) →