Chapter 32 — Authorized offensive security and red team: reliability

Lesson map

Lesson 32 published. Map, example, EX094.

flowchart TD
  N0["Learning outcomes"]
  N1["32.1 Targets"]
  N2["32.2 Retries"]
  N3["32.3 Safe degradation"]
  N4["32.2b Back-pressure"]
  N5["32.3b Chaos in synthetic only"]
  N6["32.1b Dependency health checks"]
  N7["Practice note"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX094"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["32.1 Targets"]
  N2["32.2 Retries"]
  N3["32.3 Safe degradation"]
  N4["32.2b Back-pressure"]
  N5["32.3b Chaos in synthetic only"]
  N6["32.1b Dependency health checks"]
  N7["Practice note"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX094"]

[Learning outcomes]
    |
    v
[32.1 Targets]
    |
    v
[32.2 Retries]
    |
    v
[32.3 Safe degradation]
    |
    v
[32.2b Back-pressure]
    |
    v
[32.3b Chaos in synthetic only]
    |
    v
[32.1b Dependency health checks]
    |
    v
[Practice note]
    |
    v
[Worked example] --> [Exercise EX094]

Learning outcomes

You will (1) set availability targets for assessment tools; (2) use retries with budgets; (3) degrade safely when a dependency fails.

32.1 Targets

TrackScope deny path: high availability during engagement windows. PulseMap dashboards: best-effort. State the target; do not imply five-nines without a budget.

Stating best-effort for dashboards stops five-nines folklore. Deny path availability during windows is the promise that protects the client. Sam publishes the target next to the measured value each month. Monthly values older than five weeks are marked stale on the status page. The stale marker links to the measurement job that produced the last 99.6% figure against the 99.5% deny-path target, with the job id pre-filled so operators re-run the same query rather than inventing a new one.

32.2 Retries

Retries help flaky uploads (Chapter 6) but need a deadline and jitter. Cap total attempts so the ROE rate limit still holds.

"""Reliability of a synthetic evidence-upload step with retries (deterministic)."""
import random
random.seed(7)
P_FAIL = 0.2
def upload_attempts(max_tries):
    for t in range(1, max_tries + 1):
        if random.random() > P_FAIL:
            return True, t
    return False, max_tries
for tries in (1, 2, 3, 5):
    random.seed(7)
    ok = sum(upload_attempts(tries)[0] for _ in range(1000))
    print(f"max_tries={tries} success={ok}/1000 theory={1-P_FAIL**tries:.4f}")

Expected output:

max_tries=1 success=784/1000 theory=0.8000
max_tries=2 success=949/1000 theory=0.9600
max_tries=3 success=988/1000 theory=0.9920
max_tries=5 success=1000/1000 theory=0.9997

Deadlines and jitter keep retries from becoming stampedes. Caps protect the ROE rate budget derived earlier. Sam configures upload retries from the same config file as the ROE volume limit so they cannot drift apart, and a unit test fails if the product of caps and batch size exceeds the budget. Upload config comments must cite a real ROE clause id for the volume limit — the same id Sam used when he wrote expected attempts into the ROE before the engagement — or the lint fails the build.

32.3 Safe degradation

If FindingsDesk is down, continue deny/allow locally and queue finding drafts on disk — never fail open on scope checks.

Synthetic case — Riverstone Transit. During a FindingsDesk outage, Sam's laptop queue held 14 drafts; TrackScope keeps denying out-of-scope.

Local deny/allow with on-disk draft queues preserves safety when FindingsDesk dies. Fail-open on scope is never a degradation mode. The fourteen queued drafts in the outage story show the pattern working. Sam drills the queue replay path after every FindingsDesk release. The drill seeds fourteen synthetic drafts like the outage story, replays them, then deletes them; a non-zero remaining count fails the drill before the release is marked operable.

32.4 Errors and limits

Retries against a rate-limited API can look like abuse; coordinate budgets with the ROE.

Retry storms against a limited API look like abuse to defenders watching the same network. Budgets agreed in the ROE need to include tool retries, not only logical actions. Sam shows the expected attempt count in kickoff, and he lowers scanner concurrency when the client's monitoring channel reports rate-limit alarms. Kickoff slides that omit attempt counts are corrected before signatures. Counsel's review mailbox rejects a signature packet that lacks the expected attempt count, because that number is what makes the ROE volume clause auditable.

32.2b Back-pressure

When the draft queue grows past a threshold, stop starting new discovery methods and page the lead. Continuing to scan while findings cannot be recorded creates untracked risk. Back-pressure is a safety feature, not only a performance feature.

Untracked findings while scanning continues is silent scope risk. Stopping new discovery methods pages the lead into a conscious choice. Sam's threshold is set low enough that a laptop disk queue cannot grow overnight unnoticed. Depth alarms fire when the queue passes fifty drafts — before age alarms — because a full disk during a FindingsDesk outage would drop new drafts silently.

32.3b Chaos in synthetic only

Reliability experiments (kill pods, partition networks) run in the synthetic range. Never chaos-test during a client's signed window. Record experiment ids separately from engagement ids so metrics do not mix.

Signed windows are for authorized assessment, not experiments. Sam's chaos scripts refuse to run if an engagement id is set in the environment. CI sets a fake engagement id on purpose and expects a non-zero exit, proving the guard cannot be skipped by a forgotten export in a shell profile.

32.1b Dependency health checks

TrackScope should health-check policy storage and event sink at start of window. Starting an engagement blind and discovering a broken sink mid-scan is an operations failure the reliability chapter is meant to prevent.

Blind starts that discover a broken sink mid-scan waste the client's patience and create partial evidence. Health checks at window start are cheaper than apologies. Sam blocks the start button until policy storage and the event sink both answer. When the sink failed last month's synthetic start check, the plain-text line event-sink: connection refused was pasted into the start chat so the client contact saw why the window had not opened.

Practice note

Define error budgets for the deny path separately from best-effort UI budgets. Burning the UI budget on a fancy dashboard is acceptable; burning the deny budget on the same work is not. Publish both budgets so product pressure has a number to argue with instead of a vague plea for more reliability.

32.4b Measuring whether targets were met

An availability target is only meaningful if it is measured. TrackScope records every deny or allow decision with a timestamp, so the team can compute the share of engagement-window minutes in which the service answered within its latency budget. Last month's synthetic report showed 99.6% against a 99.5% target.

Timestamps on every decision make availability computable instead of anecdotal. Last month's 99.6% against 99.5% is the kind of ugly number Sam prefers. He charts minutes inside budget beside minutes in the window so a short outage cannot hide in averages, and the chart is captioned with the same formula the reliability target uses.

Exercises

• EX094 — Target. Write an availability sentence for TrackScope deny path. Rubric: 3 = windowed target + measure; 2 = vague high; 1 = five-nines joke; 0 = none.
• EX095 — Budget. With p=0.8 and max 3 tries, expected attempts? Rubric: 3 = 1.24 with working; 2 = 1/p; 1 = 3; 0 = none.
• EX096 — Degrade. What must never degrade to ALLOW? Rubric: 3 = scope/policy decisions; 2 = partial; 1 = findings UI; 0 = wrong.

Synthesis

Reliable denies, budgeted retries, degrade closed.

Continue studying — chapter 32 of 50

← Chapter 31 (draft) · Course chapter index · Chapter 33 (draft) →