Chapter 36 — Application and software supply chain security: performance and scale
Lesson map
Lesson 36 published. Map, example, EX106.
flowchart TD
N0["Learning outcomes"]
N1["36.1 Growth"]
N2["36.2 Incremental checks"]
N3["36.3 Safe caching"]
N4["36.4 Parallelism"]
N5["36.5 Measuring cost"]
N6["36.6 Scale of data"]
N7["Planning for growth"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX106"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["36.1 Growth"]
N2["36.2 Incremental checks"]
N3["36.3 Safe caching"]
N4["36.4 Parallelism"]
N5["36.5 Measuring cost"]
N6["36.6 Scale of data"]
N7["Planning for growth"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX106"]
[Learning outcomes]
|
v
[36.1 Growth]
|
v
[36.2 Incremental checks]
|
v
[36.3 Safe caching]
|
v
[36.4 Parallelism]
|
v
[36.5 Measuring cost]
|
v
[36.6 Scale of data]
|
v
[Planning for growth]
|
v
[Worked example] --> [Exercise EX106]
Learning outcomes
You will (1) keep security controls fast as the codebase and team grow; (2) scale SBOM and advisory checks; (3) cache safely; (4) measure the cost of controls.
36.1 Growth
Over two years ticket-web grows to 220 components and 30 developers, with 60 builds a day. Checks that took seconds now risk slowing everyone.
Growth multiplies the cost of every check, because each one now runs more often and covers more components. A check that adds a minute to each build costs an hour of waiting every day across the team. Slow checks also encourage developers to batch changes or seek exceptions, which weakens the controls they were meant to enforce. Kwame therefore treats speed as a design requirement.
36.2 Incremental checks
The dependency gate checks only components that changed in a pull request, and a nightly job checks everything. This cut median gate time from 90 to 15 seconds.
Most pull requests touch few or no dependencies, so checking only what changed avoids repeating work. The nightly full check ensures advisories published for unchanged components are still found, as the worked example on zipx will show. Together they cover everything with far less waiting during the day. If the nightly run finds something, the owner is alerted the next morning.
36.3 Safe caching
Advisory data is cached for one hour; the cache key includes the feed's version so stale data is detected. Caches must never be shared between untrusted pull-request builds and release builds, or a poisoned cache could affect releases.
Including the feed version in the cache key means a newer feed invalidates old entries automatically, so the cache cannot hide a fresh advisory for longer than an hour. Separating caches by build type matters because a pull request from a fork runs code nobody has reviewed; if that code could write into the cache used for releases, it could plant a fake 'no advisories' answer. Release builds therefore use their own cache.
36.4 Parallelism
Tests and checks run in parallel jobs; provenance generation happens once at the end.
Running independent jobs at the same time shortens the wall-clock time developers wait, without reducing what is checked. Provenance must come last because it describes the final outputs and the steps that produced them; generating it earlier would leave later steps unrecorded. Parallel jobs need separate workspaces, so one job cannot alter files another job is about to check. The pipeline runs each in its own container, which also keeps a failure in one job from leaving debris for the others.
36.5 Measuring cost
Kwame measures each control's time: tests 6 minutes, static analysis 2, gate 15 seconds, signing 20 seconds, verification 12 seconds. Security controls are under 10% of CI time.
These figures answer the common complaint that security slows builds: most CI time goes to tests, which the team would run anyway. Measuring each control separately also shows where effort would pay off. Speeding up the gate further would save little, while splitting the test suite would save minutes. Numbers end arguments that impressions start. Kwame repeats the measurement each quarter.
36.6 Scale of data
SBOMs for all releases are stored in a searchable index so a question like "which releases contain zipx 1.2.x?" answers in seconds.
Without an index, the same question would mean opening every stored SBOM in turn, which takes long enough to delay a response. The index stores component names and versions for each release, linked to the release digest, so a query returns both which releases contain a component and where those releases run. Wildcard searches over a minor series help when an advisory covers a range of versions.
Planning for growth
Kwame projects build volume for the next two years using the growth rates from chapter 1 and sizes the CI capacity accordingly. He also sets a budget: security controls should stay under 10% of CI time. When a new check is proposed, its time cost is measured in staging first, and if it would break the budget, the team looks for ways to make it incremental before adding it.
Worked example
When a new advisory affects zipx, the index search returns 14 releases in 2 seconds; only the current one is deployed. The team upgrades and confirms no other environments run older releases.
Synthetic case. Sizes and timings are synthetic.
Errors and limits
Speed optimisations can introduce gaps, such as skipped checks on unchanged components; the nightly full check covers this.
Exercises
- EX106 — Incremental check. How was gate time reduced? Rubric: 3 = check changed components; nightly full; 2 = partial; 1 = vague; 0 = none.
- EX107 — Cache rule. Which caches must never be shared? Rubric: 3 = untrusted PR builds and release builds; 2 = partial; 1 = vague; 0 = none.
- EX108 — Control cost. What share of CI time do security controls take? Rubric: 3 = under 10%; 2 = partial; 1 = vague; 0 = none.
Synthesis
Scale checks with incremental work, safe caching and measured cost.
Continue studying — chapter 36 of 50
← Chapter 35 (draft) · Course chapter index · Chapter 37 (draft) →