Chapter 48 — Evidence dossier and acceptance tests

Chapter 48 — Evidence dossier and acceptance tests

Lesson map

Lesson 48 published. Map, example, EX095.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["48.1 Purpose"]
  N2["48.2 Structure"]
  N3["48.3 Standards for adequate proof"]
  N4["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4
  N4 --> EX["Worked example"]
  EX --> Q["Exercise EX095"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["48.1 Purpose"]
  N2["48.2 Structure"]
  N3["48.3 Standards for adequate proof"]
  N4["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX095"]

[Learning outcomes and connection]
    |
    v
[48.1 Purpose]
    |
    v
[48.2 Structure]
    |
    v
[48.3 Standards for adequate proof]
    |
    v
[Chapter synthesis]
    |
    v
[Worked example] --> [Exercise EX095]

Learning outcomes and connection

After this chapter you can (1) assemble a verifiable evidence dossier that maps every approved requirement to test results and artefacts; (2) record exceptions and their dispositions; (3) obtain and record reviewer sign-off; and (4) flag every requirement without adequate proof. It builds on Chapters 5, 16, 21, 44–47 and prepares Chapters 49 and 50.

48.1 Purpose

The evidence dossier answers one question for each requirement: what evidence shows that the built system satisfies it, and how strong is that evidence? It is the engineering counterpart of an audit file: someone who did not build the system should be able to verify the claims from the dossier and the repository alone. In the extract below, a reviewer can follow R1 to load test LT-3 and commit a41f… and rerun it without asking the student. Where that verification fails, the dossier entry is the place to say so plainly.

48.2 Structure

1. Index: requirement ID, statement, verification method, test IDs, result, artefact links, status (met / met with exception / not met / not verified), reviewer.
2. Artefacts: test reports with run IDs and commit hashes; logs; trial results (Chapters 46–47); analysis notes; inspection records.
3. Exceptions: each deviation — a requirement partly met, a test waived — with justification, risk, owner and disposition (accepted, fix scheduled, requirement changed through change control).
4. Open defects with severity and link to requirements.
5. Sign-off: reviewer names, independence, scope reviewed, date and any conditions.

Artefacts are identified by hash and stored immutably for the dossier's version, so the dossier itself is tamper-evident — the same technique as Chapter 19.

48.3 Standards for "adequate proof"

Evidence is adequate when it tests the requirement as stated (not a weaker version), under representative conditions, reproducibly, and when negative cases are covered where the requirement is a prohibition. A requirement verified only by a demonstration once, or only by a unit test of a component, may be "verified" in form but not in substance. Mark such cases explicitly.

In the dossier extract below, R9 shows the difference between form and substance. The retention configuration was inspected and a simulated-time test passed, but 400 days of real behaviour could not be observed in the project. The status is therefore "met with exception E1", not "met". R7, a prohibition, counts as adequate only because AT-07c tests denial of containment as well as reads.

48.4 Worked example (synthetic): dossier extract

Exceptions: E1 — 400-day retention cannot be observed within the project; accepted with periodic verification procedure defined. Flags: R11 lacks evidence (test not written; owner student, due before defense); R15 not met (defect D9 open, high; mitigation: admin view disabled in delivered configuration; requirement remains not met until fix verified).

Sign-off: two independent reviewers reviewed R1–R16; conditions: R11 and R15 to be resolved or presented as open in the defense.

48.5 Errors and limits

• Status "met" without links to reproducible artefacts.
• Exceptions buried in prose.
• Hiding unverified requirements.
• Sign-off by people who built the system.

The dossier extract shows the honest alternative to each error. R1's "met" links to LT-3 with three runs and a commit hash. E1 is its own entry with a disposition, not a footnote. R11 is shown as not verified and R15 as not met with defect D9 open. The sign-off names two reviewers independent of the build, with conditions on R11 and R15.

Exercises — chapter 48

EX095 — Flag requirements. Objective: judge adequacy. Inputs: R16 "the system shall resist alert floods without losing high-severity alerts", evidenced by a demonstration in which 1,000 low-severity alerts were sent and one high-severity alert appeared in the queue. Steps: judge adequacy and specify adequate evidence. Expected evidence: judgement and test design. Rubric: 3 = inadequate (single demonstration, small flood, one alert); proposes repeated load test at realistic multiples with many high-severity alerts measured for loss and latency; 2 = partial; 1 = accepts; 0 = none.

EX096 — Dossier tamper-evidence. Objective: protect the dossier. Inputs: the dossier index and artefacts. Steps: design how artefact integrity and dossier version are protected and verified by a reviewer. Expected evidence: design. Rubric: 3 = hashes of artefacts in index, signed or externally anchored index version, commit hashes, verification procedure; 2 = partial; 1 = vague; 0 = none.

Chapter synthesis

The evidence dossier maps every requirement to reproducible, adequate evidence, records exceptions and defects openly, protects its own integrity and carries independent sign-off — and flags plainly what is not yet proven.

Continue studying — chapter 48 of 50

← Chapter 47 · Course chapter index · Chapter 49 →

Continue to the next authored lesson.