Chapter 31 — Authorized offensive security and red team: implementation II
Lesson map
Lesson 31 published. Map, example, EX091.
flowchart TD
N0["Learning outcomes"]
N1["31.1 Upsert"]
N2["31.2 Validate on write"]
N3["31.3 API versioning"]
N4["31.2b Partial updates"]
N5["31.3b Bulk import"]
N6["31.1b Optimistic concurrency"]
N7["Practice note"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX091"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["31.1 Upsert"]
N2["31.2 Validate on write"]
N3["31.3 API versioning"]
N4["31.2b Partial updates"]
N5["31.3b Bulk import"]
N6["31.1b Optimistic concurrency"]
N7["Practice note"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX091"]
[Learning outcomes]
|
v
[31.1 Upsert]
|
v
[31.2 Validate on write]
|
v
[31.3 API versioning]
|
v
[31.2b Partial updates]
|
v
[31.3b Bulk import]
|
v
[31.1b Optimistic concurrency]
|
v
[Practice note]
|
v
[Worked example] --> [Exercise EX091]
Learning outcomes
You will (1) add idempotent finding upserts; (2) enforce schema validation on write; (3) version API responses.
31.1 Upsert
PUT /findings/{id} with full body; same id twice yields one row. Dedup fingerprints (Chapter 9) propose ids, humans confirm.
"""Deduplicate synthetic findings by a stable fingerprint (asset, weakness class, location)."""
import hashlib
raw = [
("web-01", "missing-header", "/login"),
("web-01", "missing-header", "/login"),
("web-01", "missing-header", "/account"),
("web-02", "default-credential", "admin-ui"),
("web-02", "default-credential", "admin-ui"),
("db-01", "weak-tls", "5432"),
]
seen = {}
for asset, cls, loc in raw:
fp = hashlib.sha256(f"{asset}|{cls}|{loc}".encode()).hexdigest()[:10]
seen.setdefault(fp, [asset, cls, loc, 0])[3] += 1
print(f"raw={len(raw)} unique={len(seen)}")
for fp, (a, c, l, n) in sorted(seen.items(), key=lambda x: x[1]):
print(f"{fp} {a:7} {c:19} {l:9} occurrences={n}")
Expected output:
raw=6 unique=4
f9bb6de442 db-01 weak-tls 5432 occurrences=1
66fb6f1d56 web-01 missing-header /account occurrences=1
c4bfba3b72 web-01 missing-header /login occurrences=2
296a3331f0 web-02 default-credential admin-ui occurrences=2
Full-body PUT with stable ids makes retries safe and duplicates rare. Fingerprints propose; humans confirm when collisions might hide different root causes. Sam's API docs show the double-PUT example from the fingerprint lab — six raw rows collapsing to four unique findings — as the normative client pattern, and a contract test fails the docs build if that example drifts from the running API.
31.2 Validate on write
Reject unknown fields and forbidden names; coerce severity through an enum. Validation errors return machine-readable codes for the UI.
Enums and forbidden names keep empty scores and payloads out at the edge. Machine-readable codes let the UI explain without parsing English. Sam adds a UI fixture for each new validation code in the same change. UI fixtures render the code string and the human explanation side by side. Each new code also appears in the public error catalogue used by mobile clients.
31.3 API versioning
Prefix /v1/; additive changes only within v1. Breaking changes require /v2/ and a migration note for PulseMap.
Synthetic case — Riverstone Transit. A mobile client sent
risk_score; validator rejected it — empty scores stay out (Chapter 15).
Additive v1 rules protect mobile clients from surprise breaks. Rejecting risk_score enforces the empty-score ban from fundamentals. Migration notes for v2 are reviewed by PulseMap owners before merge. Sam's compatibility tests run old clients against new servers nightly. When a break fires, the ticket names the oldest client still in the support matrix — last month that was the mobile build that still sent risk_score — so owners know which binary must move before the window.
31.4 Errors and limits
Idempotency keys must not be derived from wall-clock alone.
Wall-clock idempotency keys collide under retries in the same second. Keys must include the finding id or a client-generated nonce. Sam rejects client libraries that hash only timestamps, and the rejection message names a required key shape so authors do not invent another weak scheme. Required key shapes are illustrated with valid and invalid examples in the error body. Invalid examples in the error body are generated from the same fixture the tests use.
31.2b Partial updates
PATCH that allows severity changes without evidence references invites silent downgrades. Prefer full PUT of the validated body, or a restricted PATCH that cannot clear evidence hashes or change status to closed without a re-test pointer.
Silent severity downgrades without evidence are how risk acceptance hides. Restricted PATCH surfaces make closed-without-retest impossible. Sam prefers PUT in the official SDK and leaves PATCH for admin tooling with extra audit fields. Admin PATCH audit fields include prior severity and the actor who changed it.
31.3b Bulk import
Bulk finding import from scanners must pass the same schema validation as interactive create, including forbidden-field checks and fingerprint idempotency. A separate "trusted import" path is how payloads sneak into the store.
Sam's import job logs the scanner id as evidence, never as a reason to skip checks. Skipping checks for a scanner brand is filed as a wontfix with a link to this rule. Imports that fail validation are quarantined with the scanner's file kept as received, so the vendor can be shown exactly which field was rejected. The same fingerprint arriving twice updates the existing finding instead of creating a duplicate.
31.1b Optimistic concurrency
Use ETags or version numbers on findings so two remediators do not clobber each other. A lost update that clears a re-test pointer looks like a mysterious reopen later.
ETags make lost updates visible as conflicts instead of mystery reopens. Clearing a re-test pointer in a clobber is a safety bug. Sam's UI forces a reload-and-merge when a 412 arrives. Merge tools never auto-clear re-test pointers when resolving conflicts. Automated merge bots are disabled on findings paths for that reason, and a conflict that touches a re-test pointer requires a second human click that shows the prior pointer value before it can be cleared.
Practice note
Bulk deletes are forbidden without dual control. Soft-delete with reason is the default path; hard purge follows retention only. API clients that issue DELETE in a loop during a buggy retry get their credentials revoked automatically after a threshold — another place where reliability and safety meet.
Document the upsert semantics in the OpenAPI description so mobile clients do not invent their own retry ids.
31.4b Migration of stored findings
When the findings schema gains a field, existing FindingsDesk records need a default or a backfill. Sam writes the migration to run on a copy first, compares record counts and checksums before and after, and keeps the old schema readable for one release.
Copy-first migrations with checksum compares catch dropped rows before production. Keeping the old schema readable for one release cushions slow clients. Sam stores before and after checksums in the migration ticket, and the migration checklist refuses done status until both fields are non-empty and equal in record count to the pre-copy.
Exercises
- EX091 — Upsert. Why is POST-without-id risky for retries? Rubric: 3 = duplicates on retry; 2 = vague; 1 = wrong; 0 = none.
- EX092 — Reject. Give a JSON body that must 400. Rubric: 3 = contains payload/password/risk_score etc.; 2 = weak; 1 = valid body; 0 = none.
- EX093 — Version. Is renaming reason_code → reason a v1-safe change? Rubric: 3 = no, breaking → v2; 2 = hesitant; 1 = yes; 0 = none.
Synthesis
Idempotent writes, strict schema, explicit versions.
Continue studying — chapter 31 of 50
← Chapter 30 (draft) · Course chapter index · Chapter 32 (draft) →