◇ OpenCyber Academy
Engineering · Chapter 48 · English

Chapter 48 — Evidence dossier

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 48 — Evidence dossier

Lesson map

Lesson 48 published. Map, example, EX142.

flowchart TD
  N0["Learning outcomes"]
  N1["48.1 Structure"]
  N2["48.2 Evidence types"]
  N3["48.3 Integrity"]
  N4["48.4 Mapping"]
  N5["48.5 Limitations"]
  N6["48.6 Readability"]
  N7["Failures included"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX142"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["48.1 Structure"]
  N2["48.2 Evidence types"]
  N3["48.3 Integrity"]
  N4["48.4 Mapping"]
  N5["48.5 Limitations"]
  N6["48.6 Readability"]
  N7["Failures included"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX142"]

[Learning outcomes]
    |
    v
[48.1 Structure]
    |
    v
[48.2 Evidence types]
    |
    v
[48.3 Integrity]
    |
    v
[48.4 Mapping]
    |
    v
[48.5 Limitations]
    |
    v
[48.6 Readability]
    |
    v
[Failures included]
    |
    v
[Worked example] --> [Exercise EX142]

Learning outcomes

You will (1) compile an evidence dossier for the engineering work; (2) trace requirements to evidence; (3) include limitations; (4) prepare it for assessment and audit.

48.1 Structure

The dossier follows the requirement trace: for each of R1 to R6 and the capstone criteria, the design element, verification evidence, operational evidence and remaining risk.

Following the requirements means an assessor can start from any requirement and find everything about it in one place, instead of searching through test folders. Remaining risk is included for each, so the dossier does not imply that meeting a requirement removes all risk. The capstone criteria get the same treatment, with the creation tests, timings and developer ratings linked to their sources and the date each was collected.

48.2 Evidence types

Test results with dates and policy versions, configuration queries, measurement reports (coverage, latency, applied study), trial results, review logs, survey results and decision records.

Different types answer different questions. Tests show controls work at a point in time, configuration queries show what is set now, measurements show effect and cost, trial results show recovery under stress, and survey and review records show how people experienced the system. Each item records when it was produced and by what, so an assessor can tell generated evidence from notes written later. Decision records explain choices that no test could show.

48.3 Integrity

Evidence files are stored in a repository with a manifest of SHA-256 hashes so assessors can confirm nothing changed after submission.

The manifest lists every file with its hash and is itself committed to the repository, so the history shows when each piece of evidence was added. An assessor can rehash any file and compare. If evidence must be corrected after submission, a new file and a new manifest entry are added and the change is explained, rather than the original being replaced. That keeps the record of what was first submitted.

48.4 Mapping

A table maps evidence to CSA CCM v4 domains and to the Pod Security Standards and CIS Benchmark sections used.

Mapping to recognised frameworks helps auditors who think in those terms find what they need without learning the project's own requirement numbers. Each row names the control objective, the evidence files and any gap. Where evidence only partly meets a framework control, the table says so, as with key rotation records before they were automated. Elena cites the framework versions used, since mappings to an older version may not hold for a newer one.

48.5 Limitations

One organisation; synthetic data in trials; short post-rollout period; some controls depend on provider features that could change.

Each limitation is stated with its effect on the claims. One organisation means results may not transfer. Synthetic trial data shows controls working but not how real attackers would behave. A short period after rollout cannot show whether results hold over years. Provider features such as identity federation could change, so the dossier lists which controls depend on them and what would need rechecking if they did.

48.6 Readability

A two-page summary leads, with links into detail.

Most readers, including managers and the regulator, will read only the summary, so it must stand on its own: the problem, what was built, the main results with figures, the limits and where to find the evidence. Each claim links to the relevant part of the trace. Elena asks Rosa to read the summary without the rest and say what she understood. Rosa's summary of it, compared with Elena's intended message, shows where the wording misleads.

Failures included

The dossier includes the failures found along the way: the Baseline label error, the alert severity mistake, the canary-caught template error and the workflow permission issue found in review. Each shows how it was found and fixed. Assessors value these because they show the verification process works. Elena also includes a short section listing evidence she could not obtain, such as long-term incident data, and why.

Assessor access

Assessors receive read-only access to the evidence repository and the manifest, so they can check any file against its hash themselves. Elena walks one assessor through the trace for R2 in a short meeting.

Worked example

An assessor test-reads the dossier and asks for evidence that exceptions expire. Elena adds the exception log extract and a test showing an expired exception causing rejection, closing the gap.

Synthetic case. Dossier contents are synthetic.

Errors and limits

A dossier that only shows successes is less credible; include failures and fixes.

Exercises

Synthesis

Trace every claim to evidence and show the failures too.

Continue studying — chapter 48 of 50

← Chapter 47 (draft) · Course chapter index · Chapter 49 (draft) →