Chapter 48 — Evidence dossier
Lesson map
Lesson 48 published. Map, example, EX142.
flowchart TD
N0["Learning outcomes"]
N1["48.1 Purpose"]
N2["48.2 Structure"]
N3["48.3 Evidence types"]
N4["48.4 Gaps"]
N5["48.5 Usability"]
N6["48.6 Maintenance"]
N7["Audience needs"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX142"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["48.1 Purpose"]
N2["48.2 Structure"]
N3["48.3 Evidence types"]
N4["48.4 Gaps"]
N5["48.5 Usability"]
N6["48.6 Maintenance"]
N7["Audience needs"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX142"]
[Learning outcomes]
|
v
[48.1 Purpose]
|
v
[48.2 Structure]
|
v
[48.3 Evidence types]
|
v
[48.4 Gaps]
|
v
[48.5 Usability]
|
v
[48.6 Maintenance]
|
v
[Audience needs]
|
v
[Worked example] --> [Exercise EX142]
Learning outcomes
You will (1) assemble evidence that the system meets its requirements; (2) organise evidence by requirement; (3) include test results, reviews and trials; (4) make the dossier useful to assessors and operators.
48.1 Purpose
The dossier shows assessors and Coastline that the capstone and the pipeline controls meet their requirements.
Assessors need to judge the capstone against academic criteria, while Coastline needs confidence to operate the verifier and rely on the controls around it. One dossier serves both, because both need the same thing: each claim backed by evidence that someone else can check. Without the dossier, the evidence would be spread across repositories, reports and emails, and nobody could easily confirm that everything had been tested.
48.2 Structure
For each requirement: statement, implementation, verification method, evidence links and status. Sections cover the verifier, the pipeline controls it relies on and the operational procedures.
Repeating the same fields for every requirement makes the dossier easy to scan and makes missing evidence obvious. The verification method says how the requirement was checked; the evidence links lead straight to the results. Grouping by system part follows how operators think about the service. A status of met, partly met or not met is chosen from evidence, never from intent.
48.3 Evidence types
Test reports (chapter 46), design review records (45), failure trial results (47), validation reports (38) and metrics (29).
Different kinds of evidence answer different questions. Tests show the verifier behaves correctly in controlled cases; review records show the design was challenged and improved; the trial shows behaviour under failure; validation reports show the controls hold on running systems; metrics show performance over time. Together they give a fuller picture than any one type could, and an assessor can follow whichever kind best answers their question. Each item carries a date, so readers can tell how recent it is.
48.4 Gaps
Two requirements are partly met: monthly reports exist but are not yet reviewed by management; alert response is tested in staging only. Both are stated with plans.
Stating gaps openly makes the rest of the dossier more credible, because readers can see the author is not hiding weaknesses. Each gap has an owner and a date, which turns it from an admission into a plan. Management review of the reports is scheduled for the next quarter; a production test of alert response needs an agreed window with operations.
48.5 Usability
The dossier begins with a one-page summary table.
The table lists each requirement with its status and a link to the detail, so a reader can see the overall picture in a minute and go deeper where they need to. Assessors usually sample a few requirements; the summary helps them choose. Operators use it to find the parts that matter to them. Partly met items are marked clearly in the table rather than buried in the detail. The table is generated from the detailed entries, so it cannot drift from them, and it shows the date of the last update at the top.
48.6 Maintenance
The dossier is versioned; after handover, the platform team updates it.
A dossier that stops being updated becomes misleading, since the system and its evidence change over time. Versioning lets readers see what changed and when. The platform team adds new test results, trial outcomes and validation reports as they arrive, and closes the two gaps when their plans are complete. A yearly review checks that every link still works and every status is still accurate, and the dossier is included in the material for Coastline's annual security review.
Audience needs
Assessors want traceability; operators want procedures and contacts. The dossier therefore has two entry points: a requirement table for assessors and an operations section with runbooks, alert meanings and escalation contacts. Kwame tests each entry point with a real reader from that group and fixes the places where they got lost. He also adds a glossary for terms like provenance and transparency log, which operators found unfamiliar.
Review of the dossier
Before submission, Ingrid and the tutor check that every link works and every claim matches its evidence. Two links point to outdated test reports and are updated. A second reader checks the summary table against the detailed sections, finding one status shown as met that was only partly met; it is corrected.
Format
The dossier is plain text with links, readable without special tools.
Worked example
The assessor samples three requirements and follows the links to evidence in under ten minutes. She notes the honest gap statements as a strength.
Synthetic case. The dossier is synthetic.
Errors and limits
Evidence ages; keep dates on everything.
Exercises
- EX142 — Requirement entry. What does each requirement entry contain? Rubric: 3 = statement, implementation, method, evidence, status; 2 = four; 1 = two; 0 = none.
- EX143 — Partial items. Which two requirements were partly met? Rubric: 3 = management review of reports; production alert response; 2 = one; 1 = vague; 0 = none.
- EX144 — Assessor finding. What did the assessor note? Rubric: 3 = traced in 10 min; honest gaps a strength; 2 = partial; 1 = vague; 0 = none.
Synthesis
An evidence dossier links each requirement to proof and states gaps openly.
Continue studying — chapter 48 of 50
← Chapter 47 (draft) · Course chapter index · Chapter 49 (draft) →