Chapter 38 — Application and software supply chain security: validation
Lesson map
Lesson 38 published. Map, example, EX112.
flowchart TD
N0["Learning outcomes"]
N1["38.1 What validation means"]
N2["Lab — Lab — provenance policy"]
N3["38.2 Lab result"]
N4["38.3 Investigating denials"]
N5["38.4 Evidence for auditors"]
N6["38.5 Continuous validation"]
N7["Validation coverage"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX112"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["38.1 What validation means"]
N2["Lab — Lab — provenance policy"]
N3["38.2 Lab result"]
N4["38.3 Investigating denials"]
N5["38.4 Evidence for auditors"]
N6["38.5 Continuous validation"]
N7["Validation coverage"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX112"]
[Learning outcomes]
|
v
[38.1 What validation means]
|
v
[Lab — Lab — provenance policy]
|
v
[38.2 Lab result]
|
v
[38.3 Investigating denials]
|
v
[38.4 Evidence for auditors]
|
v
[38.5 Continuous validation]
|
v
[Validation coverage]
|
v
[Worked example] --> [Exercise EX112]
Learning outcomes
You will (1) validate that deployed artifacts meet provenance policy; (2) check records against policy rules; (3) investigate denials; (4) show evidence to auditors.
38.1 What validation means
Validation checks the real system against requirements: are only artifacts built by the approved builder, from the approved repository, from reviewed commits, running?
Validation differs from verification at deployment. Deployment checks each artifact once, as it goes out; validation looks at what is actually running at a given moment, which can differ if someone copied files by hand or if a server was missed. The question is answered from evidence, not from what the pipeline intended. That makes it a check on the pipeline itself as well.
Lab — Lab — provenance policy
"""Check synthetic build provenance records against a deployment policy."""
policy = {"builder": "ci.coastline.example/hosted-runner", "repo": "git.coastline.example/ticket-web"}
records = [{"artifact": "ticket-web-3.2.0", "builder": "ci.coastline.example/hosted-runner", "repo": "git.coastline.example/ticket-web", "commit_reviewed": True},
{"artifact": "ticket-web-3.2.1", "builder": "laptop-kd", "repo": "git.coastline.example/ticket-web", "commit_reviewed": True},
{"artifact": "ticket-web-3.2.2", "builder": "ci.coastline.example/hosted-runner", "repo": "git.coastline.example/ticket-web", "commit_reviewed": False}]
for r in records:
issues = [k for k in policy if r[k] != policy[k]] + ([] if r["commit_reviewed"] else ["unreviewed commit"])
print(f"{r['artifact']:16} {'ALLOW' if not issues else 'DENY: ' + ', '.join(issues)}")
Run with python3 lab_provenance.py. Verified output:
ticket-web-3.2.0 ALLOW
ticket-web-3.2.1 DENY: builder
ticket-web-3.2.2 DENY: unreviewed commit
38.2 Lab result
ticket-web-3.2.0 is allowed. ticket-web-3.2.1 is denied because it was built on "laptop-kd", not the approved hosted runner. ticket-web-3.2.2 is denied because its commit was not reviewed.
The policy compares exact strings, so a builder name that differs by one character, such as a similar-looking hostname, is denied. Each denial lists every failing rule, so 3.2.2 would show both problems if it also had the wrong builder. The repository check passes for all three records, since each came from the approved repository.
38.3 Investigating denials
3.2.1 was built during the earlier period before all builds moved to CI; it is retired. 3.2.2 came from an emergency change whose after-the-fact review had not been recorded; the review is completed and the release rebuilt.
Both denials had innocent causes, but the investigation was still needed: the same output would appear if an attacker had built a release elsewhere or slipped in an unreviewed commit. Retiring 3.2.1 removes an artifact the team cannot vouch for. Rebuilding 3.2.2 after review produces a new artifact with provenance showing a reviewed commit.
38.4 Evidence for auditors
A monthly report lists every running artifact and its policy result, with links to provenance. This supports SSDF PS.2 and PS.3 (archive and protect each release).
Auditors want to see controls working over time, not on one day chosen by the team. A monthly report generated automatically gives that history without extra effort. Links to provenance let an auditor pick any artifact and follow it back to source, builder and review. Keeping the reports for several years supports later questions about past releases. PS.3 asks that releases be archived and protected, which the write-once storage from the implementation work covers.
38.5 Continuous validation
The check runs hourly against running servers, not only at deployment.
Checking only at deployment would miss changes made afterwards, such as the manual file copy found by the operations alert. Hourly checks narrow the window in which an unapproved artifact can run unnoticed. A failure raises an alert to the on-call engineer and is added to the monthly report. The check reads digests from running servers and looks them up in the artifact store, so it needs no access to the servers' data. Hourly was chosen as a balance between load on the servers and speed of detection.
Validation coverage
Kwame checks that every production server is included in the hourly validation. A newly added server in a second region was missing because it was not yet in the inventory list. The check now reads the inventory from the same source the deployment tool uses, so new servers are covered automatically and no manual list can fall out of date.
Exceptions in validation
Any running artifact that fails policy must have an approved exception or be replaced within a week.
Worked example
The auditor asks how Coastline knows the hosted runner itself was not tampered with. Kwame explains the provider's attestations and Coastline's reliance on the provider's controls, and notes reproducible builds as a future independent check.
Synthetic case. Records are synthetic.
Errors and limits
Validation depends on trustworthy provenance; if the builder is compromised, records may be false.
Exercises
- EX112 — Lab results. Give the lab's three results. Rubric: 3 = 3.2.0 ALLOW; 3.2.1 DENY builder; 3.2.2 DENY unreviewed; 2 = two; 1 = one; 0 = none.
- EX113 — 3.2.2 fix. How was 3.2.2 resolved? Rubric: 3 = review completed; rebuilt; 2 = partial; 1 = vague; 0 = none.
- EX114 — Auditor answer. What future check did Kwame mention? Rubric: 3 = reproducible builds; 2 = partial; 1 = vague; 0 = none.
Synthesis
Validate continuously that what runs meets the provenance policy, and keep evidence.
Continue studying — chapter 38 of 50
← Chapter 37 (draft) · Course chapter index · Chapter 39 (draft) →