◇ OpenCyber Academy
Technologist · Chapter 9 · English

Chapter 9 — Evidence and measurement

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 9 — Evidence and measurement

Lesson map

Lesson 09 published. Map, example, EX017.

flowchart TD
  N0["Learning outcomes"]
  N1["9.1 Sources per fact"]
  N2["9.2 Coverage and denominators"]
  N3["9.3 Team metrics"]
  N4["9.5 Practical activity"]
  N5["9.7 Measurement as code"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX017"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["9.1 Sources per fact"]
  N2["9.2 Coverage and denominators"]
  N3["9.3 Team metrics"]
  N4["9.5 Practical activity"]
  N5["9.7 Measurement as code"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX017"]

[Learning outcomes]
    |
    v
[9.1 Sources per fact]
    |
    v
[9.2 Coverage and denominators]
    |
    v
[9.3 Team metrics]
    |
    v
[9.5 Practical activity]
    |
    v
[9.7 Measurement as code]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX017]

Learning outcomes

You will (1) choose valid sources for each cryptographic fact; (2) state coverage and denominators; (3) design readiness metrics for implementation teams; and (4) present measurements with blind spots.

9.1 Sources per fact

The negotiated TLS group is visible in the ServerHello and server logs; in TLS 1.3 the certificate is encrypted, so certificate algorithms come from configuration or endpoint inspection. SSH key exchange appears in daemon logs at verbose levels. VPN negotiation appears in appliance logs. Match each fact to the source that can actually see it.

KLS learned this when a passive sensor reported ECDSA for a service whose certificate it could not see; the value had been inferred, not observed. Each fact now carries the source class that can legitimately see it, and inferred values are labelled.

9.2 Coverage and denominators

"80 % hybrid" needs a denominator (connections? services? tier-1 services?) and coverage (which sensors, which sites). Implementers should report: services with hybrid enabled (configuration), share of connections negotiating hybrid (logs), and share of services with no evidence.

KLS's dashboard uses three numbers per flow: listeners with hybrid configured, divided by listeners in the inventory; handshakes negotiating hybrid, divided by handshakes logged; and services with no source, divided by tier-1 services. Each number names its sensors and the time window. A reader can then see that 96 % of browser handshakes and 0 % of SFTP sessions are hybrid, instead of one blended figure that hides the SFTP gap.

9.3 Team metrics

Useful metrics for implementers: share of tier-1 services offering and preferring hybrid; share of images built on approved base layers; time from approved change to deployment; number of classical-only listeners on internet-facing addresses; failed-handshake rate by cause.

Each metric drives a specific action at KLS. A tier-1 service offering but not preferring hybrid gets a configuration ticket; an image off the approved base gets a rebuild; a growing time-to-deploy prompts a look at the change process; any new classical-only internet listener is reviewed within a week; and a rise in failures by one cause, such as large ClientHellos, opens a vendor case.

9.4 Worked example

KLS's first dashboard showed "hybrid connections: 71 %". Breakdown: browser traffic to CDN 96 % hybrid (CDN logs, full coverage); API SDK traffic 40 % (gateway logs); partner SFTP 0 % (daemon logs); office VPN 0 % (config); scanners unknown. The single figure mostly measured browsers.

The breakdown also showed what each team had to do. CDN traffic needed nothing. SDK traffic needed customers to upgrade, so KLS published SDK versions that send the hybrid share by default. SFTP and VPN needed configuration and appliance work. Scanners needed a data source before any target could be set. The single figure had suggested none of this; the breakdown produced four tickets and one vendor request.

9.5 Practical activity

Design three metrics with definition, source, denominator, coverage and owner.

For each metric write the definition in one sentence, the data source and its query, the denominator, the coverage (which sites, sensors and time window), the owner and the action taken when the value moves. Example: "Share of partner SFTP sessions negotiating mlkem768x25519-sha256, from sshd logs on both SFTP hosts, denominator all completed sessions in the last 7 days, owner Platform Team; below 50 % after the notice date triggers partner outreach." Add one "unknown" metric beside it, counting partners with no session in the window.

9.6 Common errors and limits

At KLS the headline 71 % came from mixing traffic types; the passive sensor's "ECDSA" label came from a TLS 1.3 capture in which the certificate was encrypted, so it could not have been observed; and the scanner-coverage metric had no owner, so when the scanner broke nobody noticed the figure froze. The limit of this chapter's methods is that they measure configuration and negotiation, not whether recorded traffic already exists in an adversary's archive.

9.7 Measurement as code

Keep metric definitions in version control next to the scripts that compute them. When someone asks why the hybrid share changed from 62 % to 58 %, you can show whether the estate changed or the definition did — for example, a new sensor that added previously invisible office traffic to the denominator. A change in coverage should be annotated on the dashboard, because a falling number caused by better visibility is progress, not regression.

Also publish an "unknown" metric beside every coverage metric: the number of tier-1 services for which no source currently reports the relevant fact. Teams that see their unknown count every week tend to fix the feeds that would reduce it, which is how measurement coverage improves without a separate project.

Exercises

Synthesis

Measurements are only as good as their sources, denominators and coverage; implementers report scoped figures with blind spots visible.

Continue studying — chapter 09 of 30

← Chapter 08 (draft) · Course chapter index · Chapter 10 (draft) →