◇ OpenCyber Academy
Bachelor's · Chapter 15 · English

Chapter 15 — Evidence and measurement

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 15 — Evidence and measurement

Lesson map

Lesson 15 published. Map, example, EX043.

flowchart TD
  N0["Learning outcomes"]
  N1["15.1 Hashing"]
  N2["15.2 Metrics"]
  N3["15.3 Custody"]
  N4["15.4 Integrity vs truth"]
  N5["15.5 Sampling"]
  N6["15.6 PulseMap"]
  N7["15.7 Random sampling of closed tickets"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX043"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["15.1 Hashing"]
  N2["15.2 Metrics"]
  N3["15.3 Custody"]
  N4["15.4 Integrity vs truth"]
  N5["15.5 Sampling"]
  N6["15.6 PulseMap"]
  N7["15.7 Random sampling of closed tickets"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX043"]

[Learning outcomes]
    |
    v
[15.1 Hashing]
    |
    v
[15.2 Metrics]
    |
    v
[15.3 Custody]
    |
    v
[15.4 Integrity vs truth]
    |
    v
[15.5 Sampling]
    |
    v
[15.6 PulseMap]
    |
    v
[15.7 Random sampling of closed tickets]
    |
    v
[Worked example] --> [Exercise EX043]

Learning outcomes

You will (1) hash evidence; (2) choose non-vanity metrics; (3) keep custody.

15.1 Hashing

Detect accidental edits before filing.

A cryptographic hash such as SHA-256 turns a file into a short fingerprint that changes completely if even one byte changes. Sam hashes each evidence file when he captures it, and again when he attaches it to a finding. If the values differ, something touched the file in between, perhaps an image editor that re-saved it, a sync tool that converted line endings or a careless redaction on the original. Catching that before the report goes out is easy. Explaining it after a client disputes the finding is much harder, because a changed file looks the same as a doctored one.

15.2 Metrics

Assets touched, cards run, owners assigned — not raw finding counts.

A count of findings mostly measures how hard the testers looked and how they split issues into separate records. A rise from forty to sixty could mean the bank got worse, the scope grew or one weakness was logged per host instead of once. More useful measures describe the work and its effect: how many in-scope assets were actually tested, how many planned test methods were completed, how many findings have a named owner and a due date, and how long fixes take. PulseMap reports these, so a manager can tell whether a quiet quarter means safety or little testing.

15.3 Custody

Who/when/where for each artifact.

A chain-of-custody record lists every point at which evidence changed hands or location: captured by Sam on a testing laptop at 21:14, uploaded to FindingsDesk at 21:20, read by the reviewer the next morning, exported in redacted form for the report. Each entry has a person, a time and a place. If a finding is challenged months later, the record shows that the evidence presented is the evidence captured and who could have altered it. Gaps are recorded as gaps. An honest note that a file sat on a laptop for two days is better than a record that hides it.

15.4 Integrity vs truth

Matching hash ≠ live system still vulnerable.

A matching hash proves only that the evidence file has not changed since it was captured. It says nothing about whether the weakness still exists, whether the evidence was correctly understood, or whether the test was valid. A screenshot can be perfectly preserved and still show a misreading. A server that was vulnerable in March may have been patched in April. That is why TRB separates evidence integrity, checked with hashes, from finding status, checked by a retest. Report statements are dated to the moment of testing, for example that the endpoint returned another customer's data on 4 March, not that it does so now.

15.5 Sampling

Sample closed tickets for verification evidence.

Checking every closed ticket is rarely affordable, and checking only the ones that someone flags misses quiet failures. A random sample avoids both problems. Because the sample is chosen by chance, teams cannot predict which tickets will be checked, so every ticket needs evidence that the fix works, such as a retest result, a configuration diff or a scan. If the sample turns up missing evidence, the share missing in the sample estimates the share missing overall, and the interval methods used for detection rates apply to this proportion too.

15.6 PulseMap

Coverage views must show denominators.

A green tile that reads ninety per cent coverage hides the question that matters, ninety per cent of what. If PulseMap counts only the hosts in the asset inventory, and the inventory lacks a cloud account set up last quarter, the real coverage is lower than the tile suggests. Each coverage view therefore shows the count behind the percentage, such as forty-five of fifty registered web applications, and names the source of the total. Readers can then see when the denominator itself looks too small, which is often the most important finding on the dashboard.

15.7 Random sampling of closed tickets

Closed-ticket sampling is five tickets per month, chosen by RNG. Missing verification evidence reopens the ticket automatically. Sam publishes the reopen count so gaming the metric is visible.

The random draw is logged with its seed, so anyone can confirm that the five tickets were not hand-picked. Automatic reopening keeps the consequence impersonal. It is not Sam accusing a team, it is the rule applying itself. Publishing the reopen count makes improvement visible too. When one business unit's reopens fell from three a month to zero, the falling count showed its fixes were now being verified, not merely closed.

15.8 A dictionary for metric terms

A shared dictionary defines "touched", "validated" and "closed". Sam refuses PulseMap tiles that use undefined words.

The same word can mean different things to different teams. To a developer closed may mean the code was merged, to operations that it was deployed, and to the testers that a retest passed. A tile counting closed findings is meaningless until the word is fixed. Each dictionary entry gives a definition, the data field it maps to, an example and a counter-example. When a new tile is proposed, its labels are checked against the dictionary before it is published.

15.9 What counts as touched

"Touched" means a method card finished with evidence, not a ping. Sam's dictionary fight on that word was worth a week.

Under the loose definition, a host answered a network sweep and so counted as covered. Under the strict one, a planned test method was completed against it and left evidence. The difference was large. Coverage measured by sweeps was close to one hundred per cent, while coverage measured by completed methods was about sixty. The strict definition was less flattering and far more useful, because it showed which assets had never really been tested.

15.10 A green tile with two data points

PulseMap's tile subtitle "n=2" next to a green check teaches more than a lecture on sample size. Sam deliberately leaves a small-n green tile in the training environment to provoke questions.

The tile shows that both of two sampled tickets had verification evidence, which is a hundred per cent with almost no information. Learners who see it usually start by saying the control is working, and then someone notices the n. The discussion that follows covers what two observations can support, how wide the interval is and what tile design would prevent the misreading. That last question led to the rule that tiles with very small samples cannot turn green.

15.11 Signing off the definitions

Dictionary fight ended when CISO signed the definitions. Sam frames the signed PDF in the team wiki. Words are controls.

The signature settled the argument by making the definitions bank policy rather than the testing team's preference. Teams that disagreed could still propose changes, but through a review with a version number, not by quietly using their own meaning on a dashboard. Each PulseMap tile now cites the dictionary version it uses. When a definition changes, old and new figures are not plotted on the same line without a marker, so a jump caused by redefinition is not mistaken for real change.

15.12 Definitions on paper

Signed metric definitions hang next to the team coffee machine — paper still works.

A printed page on the wall is seen by people who would never open the wiki, including visitors from other teams who come to ask what a number means. Its main use is in quick conversations. When someone says a finding is closed, a colleague can point to the definition and ask whether the retest passed. The page carries its version number and date, so an out-of-date printout is easy to spot and replace when the signed version changes.

15.13 A yellow state for undefined terms

Undefined-word tiles show a yellow "define me" state instead of green. Yellow is harder to ignore.

A tile using a term missing from the dictionary still shows its number, but it cannot be green, because nobody can say what green would mean. The yellow state names the undefined word and links to the form for proposing a definition. This avoids two worse options. Hiding the tile loses information, and showing it green gives false reassurance. In practice most yellow tiles are fixed within a week, usually by mapping the label to an existing term.

15.14 Measurement office hours

Measurement office hours run Thursdays. Anyone may bring a tile that feels misleading. Sam's most common fix is adding n to the subtitle or changing green to yellow. He tracks how many tiles change colour each month as a quality metric of the metrics.

Office hours give people a low-stakes way to question a number without accusing whoever built the tile. Many complaints turn out to be about labels rather than data. A tile reading incidents may really be counting alerts, and the fix is a better title. Tracking colour changes shows whether the dashboard is becoming more honest over time. A month with no changes either means the tiles are good or that people have stopped coming.

Worked example (synthetic TRB)

Sam refuses a mismatched screenshot hash and re-captures.

Synthetic case — Riverstone Transit Bank (TRB). The bank, Sam Ito, TrackScope/FindingsDesk/PulseMap and all figures are invented for teaching.

Errors and limits

This book grants no authority to test real banks or payment systems. Labs use documentation-range addresses and synthetic data only; exploit code is never provided.

Exercises

Synthesis

Evidence is hashed and attributed; metrics carry denominators.

Continue studying — chapter 15 of 40

← Chapter 14 (draft) · Course chapter index · Chapter 16 (draft) →