Chapter 36 — Authorized offensive security and red team: performance and scale
Lesson map
Lesson 36 published. Map, example, EX106.
flowchart TD
N0["Learning outcomes"]
N1["36.2 Token buckets"]
N2["36.3 Canaries"]
N3["36.2b Fairness among methods"]
N4["36.3b Latency budgets"]
N5["36.1b Warm-up and cool-down"]
N6["Practice note"]
N7["36.4b Load tests that respect the client"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX106"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["36.2 Token buckets"]
N2["36.3 Canaries"]
N3["36.2b Fairness among methods"]
N4["36.3b Latency budgets"]
N5["36.1b Warm-up and cool-down"]
N6["Practice note"]
N7["36.4b Load tests that respect the client"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX106"]
[Learning outcomes]
|
v
[36.2 Token buckets]
|
v
[36.3 Canaries]
|
v
[36.2b Fairness among methods]
|
v
[36.3b Latency budgets]
|
v
[36.1b Warm-up and cool-down]
|
v
[Practice note]
|
v
[36.4b Load tests that respect the client]
|
v
[Worked example] --> [Exercise EX106]
Learning outcomes
You will (1) size scanners under ROE rate limits; (2) apply token buckets; (3) measure client impact with a canary.
36.1 Rate limits are first-class
Throughput goals that ignore the ROE are bugs. Size worker pools so the token bucket is the bottleneck, not a surprise 429 storm.
"""Lab R: token-bucket limits against unbounded consumption (synthetic).
Simulates 60 seconds of requests from a normal client and a flooding client
against a model endpoint with a per-client token bucket."""
import random
rng = random.Random(4)
def simulate(rate_per_s, burst, reqs):
tokens, last, ok = burst, 0.0, 0
for t in reqs:
tokens = min(burst, tokens + (t - last) * rate_per_s); last = t
if tokens >= 1: tokens -= 1; ok += 1
return ok
normal = sorted(rng.uniform(0, 60) for _ in range(90))
flood = sorted(rng.uniform(0, 60) for _ in range(3000))
for rate, burst in ((5, 20), (2, 10), (1, 5)):
print(f"limit={rate}/s burst={burst:2d} normal_served={simulate(rate, burst, normal):3d}/90 "
f"flood_served={simulate(rate, burst, flood):4d}/3000")
Expected output:
limit=5/s burst=20 normal_served= 90/90 flood_served= 319/3000
limit=2/s burst=10 normal_served= 90/90 flood_served= 129/3000
limit=1/s burst= 5 normal_served= 62/90 flood_served= 64/3000
If the worker pool can send faster than the ROE allows, the first busy morning will break the client's fare API. Making the token bucket the bottleneck means a bug that spawns extra workers still cannot exceed the signed rate. Sam sizes pools from the bucket, then proves it with a synthetic flood that stays clipped while a normal client continues. The pool size is stored next to the bucket rates so a later engineer cannot raise workers without seeing the ROE numbers.
36.2 Token buckets
Capacity = burst; refill = sustained rate. Lab shows flood clients get clipped while normal clients proceed.
Burst capacity covers short legitimate spikes; the refill rate is the sustained promise in the ROE. The lab's flood client hitting the clip while a normal client proceeds is the behaviour Sam wants on the depot uplink. He prints both numbers in the kickoff slide next to the 5/s and burst-20 figures from the Riverstone case. Changing either number requires the same authorization path as a ROE edit, not a silent config push.
36.3 Canaries
A synthetic canary request every minute tracks latency/error rate; stop conditions fire on threshold (Chapter 33).
Synthetic case — Riverstone Transit. Bucket 5/s burst 20 kept the fare API under 5% of capacity during the window.
A canary every minute is a sensor for the stop loop, not a performance trophy. When latency or errors cross the threshold, stop fires before riders notice a slow fare purchase. Sam ties the canary's threshold to the same stop conditions already coded for deny-lists and kill-switch files.
36.4 Errors and limits
Distributed scanners need a coordinated global bucket or they N× the limit.
Five scanners each obeying a local 5/s limit are a 25/s attack from the client's point of view. A global bucket, or a single scheduler that hands out tokens, is mandatory when methods run on more than one host. Sam fails a design review that cannot name where the global counter lives. Last year's design that trusted per-host files was rejected when five jump hosts were added to the pool.
36.2b Fairness among methods
When several methods share one bucket, a noisy enumerator can starve a quiet config review. Weight or separate buckets by method class so authorized quiet work still completes inside the window.
A noisy enumerator that empties a shared bucket leaves the authenticated config review starved until the window ends. Separate buckets by method class, or weights that reserve a minimum for quiet work, keep the ROE's authorized quiet checks from being casualties of someone else's scan. Sam's scheduler shows per-class remaining tokens on the lead's dashboard.
36.3b Latency budgets
Allocate milliseconds across DNS, TCP, TLS, predicate, event write. When the budget is exceeded, shed work; do not skip the predicate. Publish the budget in the runbook so optimizations have a yardstick.
Skipping the predicate to "make the budget" is how fail-open returns under a performance ticket. Shedding discovery work is allowed; shedding the allow/deny decision is not. Sam publishes the millisecond split in the runbook so a profiler-driven change has a yardstick and a veto. A pull request that removed event-write time from the budget to "look faster" was closed with a link to this rule.
36.1b Warm-up and cool-down
Ramp traffic at window start and drain before hard stop. Instant full-rate starts look like floods to defenders and trip unrelated alerts. Encode ramp in the scheduler, not as tribal knowledge.
Instant full rate at minute zero trips flood detectors that the engagement never intended to test. Ramping in the scheduler, and draining before the hard stop, keeps Riverstone's defenders from paging on a signed window. Sam encodes the ramp as config next to the bucket rates, not as a note in chat.
Practice note
Publish a rate-limit economics note: cost of waiting versus cost of a client outage. Most arguments for raising ROE limits dissolve when the outage cost is written down. Keep the note next to the token-bucket config so the next optimization debate starts from numbers.
Separate buckets for quiet versus noisy methods so a hungry enumerator cannot starve config review work.
36.4b Load tests that respect the client
Performance tests of TrackScope and the method runners use synthetic targets only. Rates measured there become the ceiling allowed in engagements, always below the client's agreed limit, so speed improvements never translate into more traffic on client systems than was signed for.
A TrackScope build that got faster must not quietly push more packets at the fare API than Monday's ROE allowed. Sam's load-test report lists both numbers side by side before any pool size increases. If the two numbers would ever cross, the ceiling stays at the client limit and the extra speed is left unused.
Exercises
- EX106 — Size. ROE 2 req/s; each worker does 1 req/s. Max workers? Rubric: 3 = 2 (+ coordinator note); 2 = 2; 1 = many; 0 = none.
- EX107 — Lab. At 1/s burst 5, how many flood requests of 3000 were served? Rubric: 3 = 64 from lab; 2 = approx; 1 = all; 0 = none.
- EX108 — Global. Why do three uncoordinated scanners break a 5/s ROE? Rubric: 3 = 3× rate; 2 = vague; 1 = deny; 0 = none.
Synthesis
Bucket first, scale second, canary always.
Continue studying — chapter 36 of 50
← Chapter 35 (draft) · Course chapter index · Chapter 37 (draft) →