Chapter 45 — Independent review of security and safety claims
Lesson map
Lesson 45 published. Map, example, EX089.
flowchart TD
N0["Learning outcomes and connection"]
N1["45.1 Why independent review"]
N2["45.2 Claim-evidence matrix and assurance cases"]
N3["45.3 Review checklist extract"]
N4["45.5 Triage"]
N5["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX089"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["45.1 Why independent review"]
N2["45.2 Claim-evidence matrix and assurance cases"]
N3["45.3 Review checklist extract"]
N4["45.5 Triage"]
N5["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX089"]
[Learning outcomes and connection]
|
v
[45.1 Why independent review]
|
v
[45.2 Claim-evidence matrix and assurance cases]
|
v
[45.3 Review checklist extract]
|
v
[45.5 Triage]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX089]
Learning outcomes and connection
After this chapter you can (1) prepare a claim-evidence matrix for independent review; (2) structure a safety and security case with claims, arguments, evidence and defeaters; (3) organise review with genuine independence; and (4) triage findings, reproduce failures and amend unsupported claims. It builds on Chapters 15, 38 and 44 and prepares Chapters 46, 48 and 50.
45.1 Why independent review
Designers see the system they intended. Independent reviewers see the system that exists. Review is most valuable before a trial is interpreted as success, because it can reveal that the trial would not test the claims that matter. Independence means the reviewer did not design or build the component, is not managed by the person whose work is reviewed, and has access to everything needed — code, dossier, environment. Finding F2 in the review below is the case in point: AT-02b passed only because its mocked clock never reached the escalation branch.
45.2 Claim-evidence matrix and assurance cases
A claim-evidence matrix lists each claim (from requirements and from the design's own statements), the evidence offered and its type (test, analysis, inspection, demonstration), and the reviewer's judgement. An assurance case organises claims hierarchically with arguments linking them to evidence; Toulmin's argument structure — claim, grounds, warrant, qualifier, rebuttal (Toulmin, The Uses of Argument) — underlies most notations. Defeaters are conditions under which a claim would not hold ("unless the anchor key is accessible to the store administrator"); the reviewer's job is largely to find them.
45.3 Review checklist (extract)
- Is each claim stated so that it could be false?
- Does the evidence test the claim, or a weaker one (action instead of outcome, Chapter 38)?
- Are negative and failure cases tested?
- Are assumptions (threat model, independence, provider behaviour) stated and justified?
- Are results reproducible from the repository?
- Are safety constraints (Chapter 10) represented as tests?
Applied to the HVSS-style capstone, the second question produced finding F5: "complete audit trail" was offered with no reconciliation job behind it. The third produced F4, where cross-tenant denial had been tested on reads but never on containment.
45.4 Worked example (synthetic): review of an HVSS-style capstone
Findings from a two-reviewer independent review:
| ID | Finding | Severity | Response |
|---|---|---|---|
| F1 | Claim "evidence store is tamper-evident" relies on anchors computed with a key stored in the same container as the store | high | key moved to separate service; claim qualified until verified |
| F2 | AT-02b (expiry without approval) uses a mocked clock that never reaches the escalation branch | high | reproduced; test fixed; one defect found in escalation |
| F3 | Latency objective measured with 10 synthetic events | medium | rerun with 10,000 events and percentiles |
| F4 | Cross-tenant denial tested only on reads, not on containment | high | negative tests added; passed |
| F5 | Dossier says "complete audit trail"; reconciliation job not implemented | medium | claim amended to "audit trail for workflow transitions; configuration changes not yet reconciled" |
Each high finding was reproduced by the student before fixing — reproduction confirms understanding and produces a regression test. Two claims were amended rather than "fixed", which is a legitimate and honest outcome of review.
45.5 Triage
Classify findings by severity (does it invalidate a safety or security claim?), effort and dependency. Fix high-severity findings before the trial; otherwise qualify the claim. Record disagreements: if the designer disputes a finding, both views go into the record with the decision and decision-maker.
In the HVSS-style review, F1, F2 and F4 were high severity and were fixed before the trial. F5 was medium and was handled by qualifying the claim to cover workflow transitions only. F3 was medium and was rerun with 10,000 events.
45.6 Errors and limits
- Reviewers who helped build the system.
- Fixing findings without reproducing them.
- Claims left unchanged when evidence is insufficient.
- Review after the trial, when results are already interpreted.
The two-reviewer HVSS-style review in the worked example avoided each of these errors. Neither of its two reviewers had designed or built any part of the capstone. Every high finding was reproduced before it was fixed, which in F2 also uncovered a real defect in escalation. Claims F1 and F5 were narrowed rather than left standing, and the review came before the trial, so results were never read against claims that had not been examined.
Exercises — chapter 45
EX089 — Defeaters. Objective: challenge claims. Inputs: claim "the containment control never affects targets outside the case tenant". Steps: list four defeaters and the evidence that would rule each out. Expected evidence: list. Rubric: 3 = four distinct defeaters (selector resolution, tenant reassignment during queueing, provider-side group membership effects, test coverage limited to reads) with evidence; 2 = three; 1 = vague; 0 = none.
EX090 — Claim-evidence matrix. Objective: prepare for review. Inputs: five claims from your capstone or HVSS. Steps: build the matrix with evidence type, location and self-assessed strength; mark which you expect a reviewer to challenge and why. Expected evidence: matrix. Rubric: 3 = honest strengths and anticipated challenges; 2 = partial; 1 = all "strong"; 0 = none.
Chapter synthesis
Independent review tests claims against evidence before trials are interpreted. Use claim-evidence matrices and assurance arguments, hunt for defeaters, reproduce failures before fixing them and amend claims that evidence does not support.
Continue studying — chapter 45 of 50
← Chapter 44 · Course chapter index · Chapter 46 →
Continue to the next authored lesson.