Chapter 15 — Evidence and measurement
Lesson map
Lesson 15 published. Map, example, EX043.
flowchart TD
N0["Learning outcomes"]
N1["15.1 Why logs matter"]
N2["Lab — Lab — logging coverage"]
N3["15.2 Lab result"]
N4["15.3 Fixing gaps"]
N5["15.4 Trustworthy evidence"]
N6["15.5 Measures"]
N7["Evidence log"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX043"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["15.1 Why logs matter"]
N2["Lab — Lab — logging coverage"]
N3["15.2 Lab result"]
N4["15.3 Fixing gaps"]
N5["15.4 Trustworthy evidence"]
N6["15.5 Measures"]
N7["Evidence log"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX043"]
[Learning outcomes]
|
v
[15.1 Why logs matter]
|
v
[Lab — Lab — logging coverage]
|
v
[15.2 Lab result]
|
v
[15.3 Fixing gaps]
|
v
[15.4 Trustworthy evidence]
|
v
[15.5 Measures]
|
v
[Evidence log]
|
v
[Worked example] --> [Exercise EX043]
Learning outcomes
You will (1) gather evidence about cloud controls; (2) check audit logging coverage; (3) measure gaps; (4) keep evidence trustworthy.
15.1 Why logs matter
The opening case could not measure how long the registry was open because logging was off. Audit logs record who changed what and when, and are the main evidence for investigations.
Without them, the review could only say that the registry had been open at some point, not when it opened, who pulled images or whether records-api was among them. With them, the same questions would have had definite answers within an hour. That difference is why turning logging on everywhere was treated as urgent, even before the registry was fixed, and why coverage is now measured every month rather than assumed.
Lab — Lab — logging coverage
"""Check audit logging per synthetic account and region (stdlib only)."""
accounts = {"studyhub-prod": {"eu-west": True, "eu-central": True}, "studyhub-test": {"eu-west": True, "eu-central": False},
"research-sandbox": {"eu-west": False, "eu-central": False}}
gaps = [(a, r) for a, regs in accounts.items() for r, on in regs.items() if not on]
for a, r in gaps: print(f"NO AUDIT LOG: {a} / {r}")
total = sum(len(r) for r in accounts.values())
print(f"logging coverage {total-len(gaps)}/{total}")
Run with python3 lab_logging.py. Verified output:
NO AUDIT LOG: studyhub-test / eu-central
NO AUDIT LOG: research-sandbox / eu-west
NO AUDIT LOG: research-sandbox / eu-central
logging coverage 3/6
15.2 Lab result
Audit logging is missing in studyhub-test / eu-central and in both regions of research-sandbox. Coverage is 3 of 6 account-region pairs. studyhub-prod is fully covered; the gaps are in test and research accounts, which are exactly where mistakes are often made.
Test and research accounts are created more often and with less ceremony than production, so a missed setting is more likely there. The opening case's missing logs were in one such account. A coverage figure for production alone would have looked perfect while hiding the very gaps that caused the review's main uncertainty, which is why Samuel reports coverage across every account and region.
15.3 Fixing gaps
An organisation-wide logging configuration turns on audit logs for every account and region, sending them to a separate log account that individual accounts cannot alter.
Central configuration removes the need to remember logging when a new account or region is added, which is how the original gap appeared. Sending logs to a separate account matters because an attacker or a careless administrator in one account cannot then delete the evidence of what they did. Samuel's lab check now confirms coverage remains complete each month.
15.4 Trustworthy evidence
Logs are stored with integrity protection and restricted access. Evidence collected for reports records source, time and hash.
Integrity protection means stored logs cannot be changed without detection, for example through a storage setting that prevents overwriting for a fixed period. Recording the hash of each extract lets Samuel show later that the figures in his report came from an unaltered file. Only the security team and auditors can read the log account directly. Samuel's evidence log follows the same pattern for every file he uses.
15.5 Measures
Logging coverage, time to fix findings and number of high-risk configurations are tracked monthly.
Each measure is defined once, with its formula and source, so that monthly figures are comparable. Coverage is account-region pairs with logging on, divided by all pairs. Time to fix is days from first detection to the checker no longer reporting the finding. The monthly report shows each measure with its trend over the previous six months. Changes to a definition are noted on the report, and earlier months are recalculated where possible, so a trend line never mixes two meanings.
15.6 Limits
Logs show actions, not intentions; missing data must be stated in reports.
In the sandbox example below, the log shows a researcher created a public bucket but not why; the friendly conversation that followed established it was a mistake. A report that says "no unusual pulls were logged" must also say which accounts were logging at the time, or readers will assume complete coverage that did not exist. Samuel's reports carry a short coverage statement near the top for that reason. Logs also miss actions taken outside the cloud platform, such as someone copying a downloaded image onto a laptop, which no cloud log can record.
Evidence log
Samuel keeps an evidence log for his project: each export or log extract received, its date, source, hash and where it is stored. When he quotes a figure in a report, the log shows where it came from. Tom later adopts the same format for the cloud team's audit preparation, because auditors ask exactly these questions.
Report gaps
Reports state gaps openly.
Worked example
After organisation-wide logging is enabled, coverage reaches 6 of 6. A month later, logs show a researcher creating a public bucket in the sandbox; the account-level block prevents it, and the log provides clear evidence for a friendly reminder rather than a guess.
Synthetic case. Accounts and events are synthetic.
Errors and limits
Logging only per account can be switched off by that account's administrators; central configuration is stronger.
Exercises
- EX043 — Lab coverage. What was coverage and where were gaps? Rubric: 3 = 3/6; test eu-central and research both; 2 = two; 1 = one; 0 = none.
- EX044 — Central logging. Why send logs to a separate account? Rubric: 3 = cannot be altered by individual accounts; 2 = partial; 1 = vague; 0 = none.
- EX045 — Sandbox event. What did logs show a month later? Rubric: 3 = attempted public bucket, blocked; 2 = partial; 1 = vague; 0 = none.
Synthesis
Logs are evidence; make coverage complete and tamper-resistant.
Continue studying — chapter 15 of 40
← Chapter 14 (draft) · Course chapter index · Chapter 16 (draft) →