Chapter 33 — Application and software supply chain security: safety and resilience
Lesson map
Lesson 33 published. Map, example, EX097.
flowchart TD
N0["Learning outcomes"]
N1["33.1 Assume compromise"]
N2["33.2 Response plan"]
N3["33.3 Known-good state"]
N4["33.4 Rider safety"]
N5["33.5 Exercise"]
N6["33.6 Lessons"]
N7["Communication"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX097"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["33.1 Assume compromise"]
N2["33.2 Response plan"]
N3["33.3 Known-good state"]
N4["33.4 Rider safety"]
N5["33.5 Exercise"]
N6["33.6 Lessons"]
N7["Communication"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX097"]
[Learning outcomes]
|
v
[33.1 Assume compromise]
|
v
[33.2 Response plan]
|
v
[33.3 Known-good state]
|
v
[33.4 Rider safety]
|
v
[33.5 Exercise]
|
v
[33.6 Lessons]
|
v
[Communication]
|
v
[Worked example] --> [Exercise EX097]
Learning outcomes
You will (1) plan for compromise of a dependency or the pipeline; (2) design response and recovery for supply-chain incidents; (3) protect riders during incidents; (4) learn from exercises.
33.1 Assume compromise
Even with good controls, a dependency or build component may be compromised. Resilience means detecting quickly, limiting damage and recovering to a known-good state.
Planning only for prevention leaves the team improvising when prevention fails. Assuming compromise changes the questions: how quickly would we notice, how many riders would be affected, and how fast could we return to a trusted release? The links between artifacts, SBOMs and provenance built in earlier chapters exist largely to answer those questions under pressure.
33.2 Response plan
The supply-chain playbook: identify affected artifacts using SBOMs and provenance; stop deployments; roll back to the last known-good signed release; rotate any credentials the compromised component could have reached; notify stakeholders.
The steps are ordered so that damage stops spreading before anything else. Finding affected artifacts first tells the team what to stop and what to roll back. Credentials are rotated because a malicious component may have read secrets available to its process, and rollback alone does not revoke stolen tokens. Each step has a named owner.
33.3 Known-good state
Because every artifact is linked to its SBOM and provenance, the team can find the last release that did not include a compromised version and redeploy it.
Rolling back to a release built before the compromise works only if that release is still stored, still signed and still compatible with the current database. Coastline's retention rules and migration practice make all three true. Without SBOMs, the team would have to guess which release is clean, and a wrong guess would leave the malicious code running while everyone believed it gone.
33.4 Rider safety
Ticket sales must continue. A fallback paper-ticket process and the previous app version keep service running.
During a supply-chain incident the safest technical step, stopping all releases, must not stop riders travelling. Riders who cannot buy a ticket may be stranded at a station or pier, which is a real harm. The paper process is tested twice a year so staff know how to use it. Keeping the previous app version available means riders whose phones have already updated can be moved back if needed, while people at stations follow the paper route. Customer messages explain the fallback in plain words.
33.5 Exercise
Coastline runs a tabletop: "a widely used package in ticket-web is reported compromised; versions 1.4.2–1.4.3 contained malicious code". Using SBOM search, the team finds that only one release included 1.4.2 and that it was deployed for two days.
A tabletop uses discussion rather than live systems, so no service is put at risk. The scenario is concrete enough that the team must run real queries against stored SBOMs. Finding that only one release was affected, and for only two days, narrows the response dramatically.
33.6 Lessons
The exercise shows that credential rotation would take a full day because tokens were scattered; the team consolidates them in the secrets manager.
Scattered tokens were invisible until the exercise asked the team to rotate them all at once. Consolidating them in the secrets manager gives one list of what exists, who uses it and how to rotate it, and makes rotation scriptable. The next exercise is planned to measure the new rotation time, so the improvement is shown with evidence rather than assumed. Exercises are worth running precisely because they find this kind of gap before a real incident does.
Communication
The playbook includes message templates for staff, riders and partners. Riders are told what happened, what Coastline did and whether they need to act. Messages avoid technical jargon and speculation; they are approved by the communications lead and Ingrid before release.
After the incident
A review within two weeks records lessons, owners and dates.
Worked example
Kwame times the SBOM search in the exercise: 4 minutes to list every affected release. Before SBOMs, the same question would have required inspecting builds by hand for hours.
Synthetic case. The exercise is synthetic.
Errors and limits
Rollback restores code but not data altered during compromise; check data integrity too.
Exercises
- EX097 — Playbook steps. List the playbook steps. Rubric: 3 = identify via SBOM/provenance; stop deploys; rollback; rotate credentials; notify; 2 = four; 1 = two; 0 = none.
- EX098 — Exercise finding. What did SBOM search show in the exercise? Rubric: 3 = one release with 1.4.2, deployed two days; 4 minutes; 2 = partial; 1 = vague; 0 = none.
- EX099 — Rotation lesson. What lesson came from credential rotation? Rubric: 3 = scattered tokens; consolidate in secrets manager; 2 = partial; 1 = vague; 0 = none.
Synthesis
Plan for compromise: find, stop, roll back, rotate and keep the service running.
Continue studying — chapter 33 of 50
← Chapter 32 (draft) · Course chapter index · Chapter 34 (draft) →