◇ OpenCyber Academy
Engineering · Chapter 48 · English

Chapter 48 — Evidence dossier

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 48 — Evidence dossier

Lesson map

Lesson 48 published. Map, example, EX142.

flowchart TD
  N0["Learning outcomes"]
  N1["48.1 Assurance argument"]
  N2["48.2 Evidence types"]
  N3["48.3 Assumptions"]
  N4["48.4 Residual risks"]
  N5["48.5 Audience"]
  N6["Practice note — counter-evidence"]
  N7["Keeping the dossier alive"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX142"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["48.1 Assurance argument"]
  N2["48.2 Evidence types"]
  N3["48.3 Assumptions"]
  N4["48.4 Residual risks"]
  N5["48.5 Audience"]
  N6["Practice note — counter-evidence"]
  N7["Keeping the dossier alive"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX142"]

[Learning outcomes]
    |
    v
[48.1 Assurance argument]
    |
    v
[48.2 Evidence types]
    |
    v
[48.3 Assumptions]
    |
    v
[48.4 Residual risks]
    |
    v
[48.5 Audience]
    |
    v
[Practice note — counter-evidence]
    |
    v
[Keeping the dossier alive]
    |
    v
[Worked example] --> [Exercise EX142]

Learning outcomes

You will (1) assemble an evidence dossier supporting the claim that Lantern and its vendor-watch extension are fit for purpose; (2) structure an assurance argument; (3) link every claim to evidence with integrity protection; (4) state residual risks and assumptions.

48.1 Assurance argument

Top claim: "Lantern delivers timely, trustworthy, lawfully collected intelligence to BEN's IT and OT defenders with acceptable risk." Sub-claims: timeliness, accuracy, TLP and privacy compliance, safety of OT recommendations, reliability, maintainability. Each sub-claim cites evidence.

Kofi drew the argument as a tree so readers could see how each sub-claim supports the top. The timeliness branch points to verification results and a quarter of production metrics; the safety branch points to the interlock tests and the design review. A reader who doubts one branch can follow it to the evidence without reading the rest.

48.2 Evidence types

Verification reports, validation results, drill records, review outcomes, metrics over a quarter, privacy assessment and partner feedback. Each item has a date, author, version and SHA-256 hash recorded in an index.

The index lets a reviewer confirm that an item has not changed since it was cited, as the worked example shows. Kofi keeps the evidence itself in a read-only store and the index in the dossier. An item without a date or author is not accepted, because nobody could judge whether it is still relevant.

48.3 Assumptions

The argument depends on assumptions: partners label TLP correctly; NTP is reliable; OT engineers follow the two-person rule. Assumptions are stated, and where possible monitored.

Kofi lists how each assumption is checked. Mislabelled TLP from partners shows up as gate rejections and partner corrections, which are counted monthly. Time synchronisation is monitored after the clock skew failure in the verification lab. Adherence to the two-person rule is checked from approval logs. An assumption that cannot be monitored is flagged as a weakness in the argument.

48.4 Residual risks

Unseen partner formats, adversary poisoning that mimics legitimate patterns, staff turnover. Each risk has an owner and acceptance by an accountable manager.

Kofi writes each risk with what would happen if it occurred and what reduces it. Poisoning that looks legitimate is the hardest, since the allow-list catches only known suppliers, a gap Kofi also concedes at the final defence in Chapter 50. Staff turnover is reduced by documentation and pairing. The accountable manager signs the acceptance, which makes clear that the risk is a business decision rather than an engineering oversight.

48.5 Audience

The dossier serves BEN's risk committee, auditors and future engineers. A two-page summary leads; the full evidence sits behind it.

Each audience reads differently. The risk committee reads the summary and the residual risks. Auditors sample evidence and check hashes. Future engineers need the assumptions and the design reasons, so they know what must stay true if they change something. Kofi asked a member of each group to read a draft and say what they could not find. Two gaps were fixed that way.

Practice note — counter-evidence

The dossier includes evidence against claims too, such as the failed drill D2 and its fix. Omitting counter-evidence makes an argument look stronger and be weaker.

Keeping the dossier alive

After go-live the dossier is updated each quarter with new metrics and drill records. Claims whose evidence has aged beyond its review date are marked as needing renewal until fresh evidence arrives.

Worked example

Kofi's index lists 37 evidence items. A reviewer picks five at random, recomputes their hashes and finds all match. One sub-claim — maintainability — rests only on a single documentation test; the reviewer asks for a second piece of evidence, and Kofi adds the technical-debt register trend.

Synthetic case. Counts are invented.

Errors and limits

An assurance argument shows reasons for confidence, not proof of security. Evidence ages; set review dates.

Exercises

Synthesis

Argue with structure, index evidence with hashes, state assumptions and residual risks, and include counter-evidence.

Continue studying — chapter 48 of 50

← Chapter 47 (draft) · Course chapter index · Chapter 49 (draft) →