Chapter 32 — Application and software supply chain security: reliability
Lesson map
Lesson 32 published. Map, example, EX094.
flowchart TD
N0["Learning outcomes"]
N1["32.1 Reliable controls"]
N2["32.2 External dependencies"]
N3["32.3 Fail closed, recover fast"]
N4["32.4 Measuring availability"]
N5["32.5 Redundancy"]
N6["32.6 Testing failure"]
N7["Error messages"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX094"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["32.1 Reliable controls"]
N2["32.2 External dependencies"]
N3["32.3 Fail closed, recover fast"]
N4["32.4 Measuring availability"]
N5["32.5 Redundancy"]
N6["32.6 Testing failure"]
N7["Error messages"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX094"]
[Learning outcomes]
|
v
[32.1 Reliable controls]
|
v
[32.2 External dependencies]
|
v
[32.3 Fail closed, recover fast]
|
v
[32.4 Measuring availability]
|
v
[32.5 Redundancy]
|
v
[32.6 Testing failure]
|
v
[Error messages]
|
v
[Worked example] --> [Exercise EX094]
Learning outcomes
You will (1) make security controls reliable; (2) avoid single points of failure in the pipeline; (3) plan for outages of external services; (4) measure control availability.
32.1 Reliable controls
A control that often fails gets switched off. The dependency gate and deployment verification must be dependable, with clear errors and quick recovery.
When a control fails often for reasons unrelated to security, developers learn to treat its failures as noise and press for an override. Each false or flaky failure therefore weakens the control. Coastline tracks failures by cause and fixes the commonest ones first, so the control mostly fails only when it has found something real. Recovery time matters as well: a failure that clears in a minute barely interrupts work, while one needing a support ticket costs hours and goodwill.
32.2 External dependencies
The pipeline depends on the public registry, the advisory feed, the transparency log and the CI provider. Each can fail. The internal mirror covers registry outages; cached advisory data covers short feed outages; verification material is cached for transparency-log outages.
Listing the dependencies makes their failures predictable. For each one, the team decides how long it can be unavailable before releases stop and what cache or fallback covers that period. A dependency with no fallback is a single point of failure and is listed as a risk until one is added.
32.3 Fail closed, recover fast
Failing closed protects integrity, but long outages block releases. Coastline sets a recovery process: if a check is unavailable for more than four hours and a critical fix must ship, Ingrid can approve a documented manual verification with two engineers.
The manual path is narrow on purpose: it needs a critical fix, a long outage and a named approver, and two engineers perform the checks by hand and record what they did. Every use is reviewed afterwards. Without this path, teams under pressure would invent their own bypasses, which would be less careful and leave no record.
32.4 Measuring availability
The team logs every control failure: cause, duration and effect. Over a quarter the gate was unavailable for 3 hours in total, all due to the advisory feed.
Three hours across a quarter is a small fraction of the time, but if it falls during an urgent release it matters a great deal. Logging the cause shows that all downtime came from one source, which makes the fix obvious. Without the log, the team would remember the outages vaguely and might blame the gate itself. The same log later shows whether the fix worked, by comparing the next quarter's downtime with this one.
32.5 Redundancy
A second advisory source is added, reducing dependence on one feed.
With two feeds, the gate keeps working if one is down, and advisories published by only one source are still seen. The sources sometimes disagree on version ranges; when they do, the gate uses the wider range and flags the difference for a person to check. Duplicates from the two feeds are merged by advisory identifier, so the earlier noise reduction is not undone. The added cost is small compared with releases blocked for hours during an outage of a single feed, and the team reviews both sources yearly to see whether each still adds value.
32.6 Testing failure
Every quarter the team simulates an outage of each external service in staging to check the fallback works.
A fallback that is never exercised may have broken without anyone knowing, as the expired cache in the worked example shows. Simulating outages in staging, by blocking access to each service in turn, proves the cache, mirror and second feed take over. Each test records how long builds and deployments continued normally and what errors appeared. Results feed into the control failure log, so planned tests and real outages can be compared over time.
Error messages
When a control fails because of an outage, the message says so plainly and links to the status page, so developers do not waste time thinking their code is wrong. Clear outage messages reduced support questions during the advisory feed outage by more than half.
Capacity
Build capacity is monitored so checks do not slow when many changes arrive at once.
Worked example
The quarterly simulation of a transparency-log outage reveals that cached verification material expired after 24 hours. The cache period is extended to 7 days with monitoring of freshness.
Synthetic case. Outages and timings are synthetic.
Errors and limits
Fallbacks add complexity; keep them simple and tested.
Exercises
- EX094 — External dependencies. List four external services the pipeline depends on. Rubric: 3 = registry, advisory feed, transparency log, CI provider; 2 = three; 1 = one; 0 = none.
- EX095 — Manual path. When can manual verification be used? Rubric: 3 = check down >4 h; critical fix; Ingrid approves; two engineers; 2 = partial; 1 = vague; 0 = none.
- EX096 — Simulation finding. What did the outage simulation reveal? Rubric: 3 = cache expired after 24 h; extended to 7 days; 2 = partial; 1 = vague; 0 = none.
Synthesis
Controls must be reliable, with tested fallbacks that keep integrity.
Continue studying — chapter 32 of 50
← Chapter 31 (draft) · Course chapter index · Chapter 33 (draft) →