Chapter 9 — Evidence and measurement | Technologist OT, IoT and cyber-physical security

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 09 published. Map, example, EX025.

flowchart TD
  N0["Learning outcomes"]
  N1["9.1 Recording evidence"]
  N2["9.2 Observation and conclusion"]
  N3["9.3 Denominators"]
  N4["9.4 Uncertainty"]
  N5["9.5 Chain of custody in a small site"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX025"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["9.1 Recording evidence"]
  N2["9.2 Observation and conclusion"]
  N3["9.3 Denominators"]
  N4["9.4 Uncertainty"]
  N5["9.5 Chain of custody in a small site"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX025"]

[Learning outcomes]
    |
    v
[9.1 Recording evidence]
    |
    v
[9.2 Observation and conclusion]
    |
    v
[9.3 Denominators]
    |
    v
[9.4 Uncertainty]
    |
    v
[9.5 Chain of custody in a small site]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX025]

Learning outcomes

You will (1) collect evidence with timestamps, sources and hashes; (2) separate what you saw from what you concluded; (3) count with denominators; and (4) present a measurement with its uncertainty.

9.1 Recording evidence

For every observation record: what was observed, where, when (with clock source), by whom and how. For exported files add a SHA-256 hash, computed with hashlib.sha256(open(path, "rb").read()).hexdigest(), so anyone can confirm the file has not changed.

Priya's entry E-014 is the pattern: timestamp 2026-03-03 10:22 with the laptop clock synced to NTP, her name, the firewall config exported from FW-1's web interface as fw1-0303.txt, and a SHA-256 hash recorded with the file.

The hash, beginning 9c1e…, lets anyone confirm later that the file has not changed since that entry.

9.2 Observation and conclusion

"Rule 9 permits any source to TCP 502 on 10.20.0.0/24 (firewall export 3 March, hash recorded)" is an observation. "Office ransomware could reach the PLCs" is a conclusion. Both belong in a report, labelled. Readers trust reports that make the line visible.

Priya's entry E-014 keeps the two on separate lines: the observation that rule 9 allows any source to TCP 502 on 10.20.0.0/24, and, on its own line, the conclusion that office hosts can reach PLC Modbus ports. A reader who disagrees with the conclusion can still rely on the observation, and can check it against the hashed export.

9.3 Denominators

"Five loggers have expired certificates" means little without "out of 40". "Two of 11 accounts belong to former staff" is clearer than "some leaver accounts exist". Percentages without counts hide small samples.

At NCS the denominators come from Priya's own tables: 2 former-staff accounts out of 11 HMI accounts, 1 expired logger certificate out of the 4 the Chapter 22 lab checks, 2 unapproved flows out of 5 conversations seen. Each count tells the reader both the size of the problem and the size of what was checked. A report that said "some accounts belong to leavers" would invite a reader to guess whether that meant 2 or 20.

9.4 Uncertainty

If you checked 10 of 40 loggers and found 2 with default passwords, the fleet rate is uncertain. Say: "2 of 10 sampled; the remaining 30 not yet checked." Avoid extrapolating "20 % of the fleet" unless the sample was random and you state the uncertainty.

At NCS, the vendor's answer that each logger has a unique password printed on its label is a claim about all 40; a sample of 10 checked on the bench tests it. If all 10 match the claim, write "10 of 10 sampled met it; 30 not checked", not "the fleet meets 5.1".

Worked example (synthetic NCS)

Priya's evidence log entry: "E-014, 2026-03-03 10:22 (laptop clock synced to NTP), Priya Nair, exported firewall config from FW-1 web UI, file fw1-0303.txt, SHA-256 starting 9c1e…, observation: rule 9 allows any → 10.20.0.0/24 TCP 502; conclusion (separate line): office hosts can reach PLC Modbus ports."

Synthetic case — Northgate Cold Storage. Evidence IDs and hashes are invented.

9.5 Chain of custody in a small site

Small sites rarely have evidence lockers. A practical minimum at NCS is an encrypted folder per investigation, a read-only copy of every original export, and a custody log listing each transfer: who handed what to whom, when and why. If the case later goes to an insurer or to police, that log is what shows the evidence was not altered between the firewall and the report. Keep the log with the evidence, not in someone's e-mail.

Priya's E-014 export is the first entry in such a log.

Errors and limits

Clocks on devices drift; note which clock a timestamp came from. A hash proves a file is unchanged since hashing, not that its content was true.

Exercises

EX025 — Observation or conclusion? Label: (a) "HMI-C had a desktop-sharing service running at 14:05"; (b) "the contractor was using it".
Show marking criteria
Rubric: 3 = observation, conclusion; 2 = with weak reasoning; 1 = reversed; 0 = none.
EX026 — Rewriting with a denominator. Rewrite "several loggers need new certificates".
Show marking criteria
Rubric: 3 = count of total with date; 2 = count only; 1 = percentage only; 0 = none.
EX027 — Reporting a partial sample. Write the sentence reporting 2 default-password loggers among 10 checked.
Show marking criteria
Rubric: 3 = 2 of 10 sampled, 30 unchecked, no extrapolation; 2 = partial; 1 = "20 % of fleet"; 0 = none.

Synthesis

Evidence becomes trustworthy when it is timestamped, hashed, counted against a denominator and kept separate from interpretation.

Continue studying — chapter 09 of 30

← Chapter 08 (draft) · Course chapter index · Chapter 10 (draft) →