Chapter 36 — Cloud and container security: performance and scale
Lesson map
Lesson 36 published. Map, example, EX106.
flowchart TD
N0["Learning outcomes"]
N1["36.1 Why measure"]
N2["Lab — Lab — admission overhead"]
N3["36.2 Lab result"]
N4["36.3 Scaling"]
N5["36.4 Network policy cost"]
N6["36.5 Optimisation"]
N7["36.6 Never trading away protection"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX106"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["36.1 Why measure"]
N2["Lab — Lab — admission overhead"]
N3["36.2 Lab result"]
N4["36.3 Scaling"]
N5["36.4 Network policy cost"]
N6["36.5 Optimisation"]
N7["36.6 Never trading away protection"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX106"]
[Learning outcomes]
|
v
[36.1 Why measure]
|
v
[Lab — Lab — admission overhead]
|
v
[36.2 Lab result]
|
v
[36.3 Scaling]
|
v
[36.4 Network policy cost]
|
v
[36.5 Optimisation]
|
v
[36.6 Never trading away protection]
|
v
[Worked example] --> [Exercise EX106]
Learning outcomes
You will (1) measure the performance cost of security controls; (2) analyse latency percentiles; (3) plan controls for growth; (4) optimise without weakening protection.
36.1 Why measure
Developers worry that admission checks and signature verification slow deployments. Measurement replaces worry with data and shows where optimisation is needed.
Without numbers, the argument tends to be settled by whoever complains loudest, and a control can be switched off to calm a team that never measured anything. Elena frames the question narrowly: how many milliseconds does admission add to one deployment, at the median and in the slow tail? A narrow question can be answered with a small, repeatable test, and the answer can be checked again after every policy change instead of being argued from memory.
Lab — Lab — admission overhead
"""Admission check overhead on synthetic deployment latencies (ms): median and p95 with and without (stdlib only)."""
import statistics
base = [180, 190, 200, 205, 210, 215, 220, 230, 240, 260, 270, 300, 310, 330, 350, 400, 420, 450, 500, 610]
withc = [b + d for b, d in zip(base, [12, 14, 15, 15, 16, 16, 17, 18, 18, 19, 20, 20, 21, 22, 24, 25, 28, 30, 35, 60])]
def p95(x): return statistics.quantiles(x, n=20, method="inclusive")[18]
print(f"median {statistics.median(base)} -> {statistics.median(withc)} ms; p95 {p95(base):.1f} -> {p95(withc):.1f} ms")
Run with python3 lab_overhead.py. Verified output:
median 265.0 -> 284.5 ms; p95 505.5 -> 541.8 ms
36.2 Lab result
Across 20 synthetic deployments, median admission latency rises from 265.0 ms without the policy checks to 284.5 ms with them, about 20 ms. The 95th percentile rises from 505.5 to 541.8 ms, about 36 ms. Deployments take minutes overall, so the overhead is negligible; the tail grows more than the median because signature lookups occasionally miss the cache.
The largest single delay in the synthetic list, 60 ms on the slowest deployment, is the kind of cache miss that pulls the 95th percentile up. Twenty samples are few for a tail estimate, so Elena treats the 36 ms as a rough figure and repeats the run before drawing conclusions.
36.3 Scaling
As meters grow (chapter 6), ingest pods and deployments increase. Policy checks are per admission, not per request, so they scale with deployment frequency, not traffic.
That distinction matters for planning. If meter numbers double, ingest traffic roughly doubles, but admission load rises only if teams deploy more often or the autoscaler adds pods more frequently. Elena therefore sizes the policy engine from deployment and scaling events per hour, which she reads from the API server's audit log, and keeps meter growth for sizing the ingest service itself. Each figure goes into the capacity plan with the date and the source it was read from.
36.4 Network policy cost
Network policy enforcement happens in the node's data path; Elena measures ingest throughput with and without policies and sees under 2% difference.
She runs the comparison on identical staging nodes with the same synthetic meter stream, alternating policy on and off so that background load affects both. Under two percent is small enough to accept, but she also notes what grows the cost: very long rule lists on a pod and frequent policy changes, which force the data path to reload rules. Keeping each namespace's policies short and grouped by label holds that cost down as the platform grows.
36.5 Optimisation
Caching signature verification results by digest cuts repeated lookups; the policy engine's replicas scale with admission load.
The cache key is the image digest, and the stored value records the verification result, the signing identity and the time of the check. An entry expires after a set period, so a revoked signing key is noticed when entries are refreshed rather than never. Replicas are added when admission queue time rises, which Elena watches as a metric, and at least two replicas always run so a single restart does not block deployments across the cluster.
36.6 Never trading away protection
Optimisations must not skip checks; caching by digest is safe because the digest identifies exact content.
A tag such as ingest:latest can point at different images from one day to the next, so a cache keyed on tags could approve an image that was never verified. A digest cannot change without the content changing. The rule Elena writes into the design notes is that an optimisation may remove repeated work, never a decision: if the engine is too slow to verify an unseen digest, the deployment waits or is rejected rather than being admitted unchecked.
Measurement method
Elena measures admission latency from the API server's own metrics in staging, running the same synthetic deployments with the policy checks enabled and disabled in alternating batches. Alternating reduces the effect of background load changing over the test. She records cluster version, node sizes and policy version with the results so the test can be repeated.
Worked example
After adding a digest cache, the synthetic p95 overhead drops to about 15 ms in a repeat test. Elena publishes the measurements to developers, which ends a long-running debate about whether the controls "slow everything down".
Synthetic case. Latencies are synthetic.
Errors and limits
Small samples give noisy percentiles; repeat measurements before deciding.
Exercises
- EX106 — Lab medians. What were the medians? Rubric: 3 = 265.0 to 284.5 ms; 2 = partial; 1 = vague; 0 = none.
- EX107 — Tail growth. Why did the tail grow more? Rubric: 3 = cache misses on signature lookups; 2 = partial; 1 = vague; 0 = none.
- EX108 — Safe cache. Why is caching by digest safe? Rubric: 3 = digest identifies exact content; 2 = partial; 1 = vague; 0 = none.
Synthesis
Measure control costs and optimise without weakening checks.
Continue studying — chapter 36 of 50
← Chapter 35 (draft) · Course chapter index · Chapter 37 (draft) →