Chapter 11 — Application and software supply chain security: verification
Lesson map
Lesson 11 published. Map, example, EX041.
flowchart TD
N0["Learning outcomes"]
N1["11.1 Verification versus configuration"]
N2["11.2 Synthetic test cases"]
N3["11.3 Recording results"]
N4["11.4 Re-verification"]
N5["11.5 Failures"]
N6["11.6 Customer evidence"]
N7["Verification schedule"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX041"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["11.1 Verification versus configuration"]
N2["11.2 Synthetic test cases"]
N3["11.3 Recording results"]
N4["11.4 Re-verification"]
N5["11.5 Failures"]
N6["11.6 Customer evidence"]
N7["Verification schedule"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX041"]
[Learning outcomes]
|
v
[11.1 Verification versus configuration]
|
v
[11.2 Synthetic test cases]
|
v
[11.3 Recording results]
|
v
[11.4 Re-verification]
|
v
[11.5 Failures]
|
v
[11.6 Customer evidence]
|
v
[Verification schedule]
|
v
[Worked example] --> [Exercise EX041]
Learning outcomes
You will (1) verify that controls work as intended; (2) test with synthetic cases; (3) record results; (4) re-verify after changes.
11.1 Verification versus configuration
Configuring a control is not the same as knowing it works. Verification checks the result: the header is actually sent, the hash check actually stops a mismatched file, the permission check actually finds writable folders. At Pinecrest this difference showed up when a template update silently dropped Referrer-Policy from 40 sites. The configuration had once been correct, but only the quarterly verification found that it no longer was. Verification asks 'does it work now?', not 'did we set it up once?'.
11.2 Synthetic test cases
For each control, Tomás prepares safe synthetic cases: a deliberately mismatched test file for the hash step, a test site with a writable code folder, and a test page loading a script from an unlisted domain to confirm CSP reports it. OWASP WSTG v4.2 describes structured testing of configuration and deployment management. Each case is harmless by design: the test file is synthetic, the writable folder sits on a test site, and the unlisted script only checks reporting. Each case has an expected result written down before the test runs.
11.3 Recording results
Each verification records control, test case, expected result, actual result, date and tester. Recording the expected result before the test prevents reading success into whatever happens. A record for the hash step might show a mismatched test file, expected MISMATCH, actual MISMATCH, with date and tester. Over six months these records added up to 31 verifications with three failures, which became Grace's evidence of maintained controls. Records are kept with the control's documentation, so anyone can see its test history at once.
11.4 Re-verification
Controls are re-verified after server template changes, application upgrades and every quarter. Template changes and application upgrades are exactly when settings tend to drift, as the dropped Referrer-Policy on 40 sites showed. Quarterly runs catch slower changes that no single event announces. The calendar Tomás keeps lists each re-verification, so another technician can run them when he is away. Each re-verification uses the same synthetic cases, so results can be compared over time and a change in outcome stands out clearly.
11.5 Failures
A failed verification is a finding with an owner and due date, like any other issue. Treating a failed verification as a finding means it enters the ticket queue with the same visibility as any other problem. The three failures in six months were each fixed within a week this way. Without an owner and due date, a failed test risks being noted and then forgotten. The failure record also notes which test found it, which helps judge whether tests are sufficient.
11.6 Customer evidence
Verification records let Pinecrest answer customer security questionnaires with evidence. When a customer's questionnaire asks whether file integrity is checked before installation, Pinecrest can point to dated verification records rather than a policy statement. The records show the hash step actually rejecting a mismatched file, which is stronger evidence than a procedure that only says what should happen. Answers cite dates and results without exposing other customers' details, and Grace reviews each questionnaire answer before it goes back to the customer.
Verification schedule
Tomás keeps a simple calendar: hash-step test monthly, permission check weekly through the nightly job, header check after every template change and quarterly in full. Each test takes minutes. The calendar is shared, so if Tomás is away another technician runs the tests and records the results. After six months the records show 31 verifications with three failures, all fixed within a week, which Grace uses as evidence that the controls are being maintained, not just configured once.
Worked example
Quarterly verification finds that the server template update had dropped Referrer-Policy from 40 sites. The test catches it within days, the template is fixed, and a check is added to the template's own tests.
Synthetic case. Tests and results are synthetic.
Errors and limits
Tests only cover cases you thought of; vary them over time.
Exercises
- EX041 — Difference. How does verification differ from configuration? Rubric: 3 = checks the result works; 2 = partial; 1 = vague; 0 = none.
- EX042 — Test cases. Name two synthetic test cases. Rubric: 3 = mismatched file, writable folder, unlisted script; 2 = one; 1 = vague; 0 = none.
- EX043 — When to re-verify. When are controls re-verified? Rubric: 3 = template changes, upgrades, quarterly; 2 = two; 1 = one; 0 = none.
- EX044 — Quarterly catch. What did the quarterly check catch? Rubric: 3 = Referrer-Policy dropped from 40 sites; 2 = partial; 1 = vague; 0 = none.
Synthesis
Verify controls with safe tests and repeat after change.
Continue studying — chapter 11 of 18
← Chapter 10 (draft) · Course chapter index · Chapter 12 (draft) →