Chapter 9 — Programming II | Engineering Authorized offensive security and red team

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 9 — Programming II

Lesson map

Lesson 09 published. Map, example, EX025.

flowchart TD
  N0["Learning outcomes"]
  N1["9.1 Schemas without payloads"]
  N2["9.2 Idempotent updates"]
  N3["9.3 Property tests"]
  N4["9.2b Migration without downtime"]
  N5["9.1b Idempotency keys from fingerprints"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX025"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["9.1 Schemas without payloads"]
  N2["9.2 Idempotent updates"]
  N3["9.3 Property tests"]
  N4["9.2b Migration without downtime"]
  N5["9.1b Idempotency keys from fingerprints"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX025"]

[Learning outcomes]
    |
    v
[9.1 Schemas without payloads]
    |
    v
[9.2 Idempotent updates]
    |
    v
[9.3 Property tests]
    |
    v
[9.2b Migration without downtime]
    |
    v
[9.1b Idempotency keys from fingerprints]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX025]

Learning outcomes

You will (1) design a finding schema that forbids exploit recipes; (2) implement idempotent updates; (3) use property tests for scope rules.

9.1 Schemas without payloads

FindingsDesk stores title, asset, weakness class, evidence reference, impact, status and timestamps. It must not store a working exploit, a live credential or a step-by-step attack. Schema validation rejects fields named payload, password or shell. Evidence lives in a sealed store referenced by hash (Chapter 48).

Rejecting those field names at the door stops a well-meaning engineer pasting a password into a ticket during a busy close-out. Evidence by hash means the ticket stays shareable while the sealed store stays locked. Sam reviews every new column proposal against the same rule: if it could hold a recipe or a secret, it does not belong in FindingsDesk.

9.2 Idempotent updates

Closing a finding twice must leave it closed; attaching the same evidence hash twice must leave one attachment. Idempotency lets retries (Chapter 6) be safe. Use a stable finding id and treat create-or-update as a single operation keyed by that id.

"""Deduplicate synthetic findings by a stable fingerprint (asset, weakness class, location)."""
import hashlib
raw = [
    ("web-01", "missing-header", "/login"),
    ("web-01", "missing-header", "/login"),
    ("web-01", "missing-header", "/account"),
    ("web-02", "default-credential", "admin-ui"),
    ("web-02", "default-credential", "admin-ui"),
    ("db-01", "weak-tls", "5432"),
]
seen = {}
for asset, cls, loc in raw:
    fp = hashlib.sha256(f"{asset}|{cls}|{loc}".encode()).hexdigest()[:10]
    seen.setdefault(fp, [asset, cls, loc, 0])[3] += 1
print(f"raw={len(raw)} unique={len(seen)}")
for fp, (a, c, l, n) in sorted(seen.items(), key=lambda x: x[1]):
    print(f"{fp} {a:7} {c:19} {l:9} occurrences={n}")

Expected output:

raw=6 unique=4
f9bb6de442 db-01   weak-tls            5432      occurrences=1
66fb6f1d56 web-01  missing-header      /account  occurrences=1
c4bfba3b72 web-01  missing-header      /login    occurrences=2
296a3331f0 web-02  default-credential  admin-ui  occurrences=2

Retries are normal when uploads fail, and without idempotency each retry would reopen work or create duplicate tickets. A stable id turns create and update into the same path, so the second call finds the existing row. The lab's fingerprint collapses six raw rows to four unique findings, which is what remediation owners want to see.

9.3 Property tests

Beyond example tests, state a property: "for every host in the deny-list and every port, allowed is False". Generate dozens of synthetic hosts and assert the property. Property tests catch the bug where a deny-list entry is ignored for ports not in a hard-coded list.

Synthetic case — Riverstone Transit. Deduplicating six raw rows left four unique findings; the two duplicates were the same missing header on /login reported by two scanners. Sam made the fingerprint (asset, weakness, location) the idempotency key.

9.4 Errors and limits

Schemas drift; version them. Property tests are only as good as the generator; include known edge cases by hand.

A schema without a version field forces clients to guess which fields exist. Generators that never emit the last address in a range will never catch a bug there. Sam keeps a small hand-written table of nasty cases next to the property suite and runs both on every change. A generator bug is treated like a product bug: fix it before the next engagement.

9.2b Migration without downtime

When the finding schema gains a required field, deploy a two-phase change: first accept and persist the field as optional while backfilling; then flip the validator to required. Engagements mid-window must not break because a mobile client is one version behind. Publish the schema version in API responses so clients can refuse to write when incompatible.

Two-phase changes keep mid-engagement clients writing safely. Backfill fills old rows before the field becomes required, so validators do not reject historical tickets. Publishing the schema version lets a mobile client refuse an incompatible write instead of corrupting a finding. Sam rehearses the flip on a staging copy with a live engagement fixture before touching production.

9.1b Idempotency keys from fingerprints

When two scanners report the same weakness, compute the fingerprint first and use it to mint or look up the finding id before insert. Humans still confirm merges when fingerprints collide across different root causes (same header missing on two unrelated apps). Log the scanner ids as evidence references, not as separate findings, so remediation tracks one ticket.

Scanners disagree on wording but often agree on asset, class and location, which is enough for a fingerprint. Collisions across unrelated apps are rare and need a human merge, which is cheaper than dozens of duplicate tickets. Storing scanner ids as evidence keeps provenance without multiplying the backlog. Sam shows the fingerprint in the ticket so owners can see why two alerts became one.

Exercises

EX025 — Schema reject. Write a validation rule that rejects a finding containing a field whose name matches (?i)payload|password|passwd|shell. Rubric: 3 = rule plus a failing example; 2 = rule only; 1 = idea; 0 = none.
EX026 — Fingerprint. From the lab, why do two /login rows collapse and two /account rows not? Rubric: 3 = same fingerprint vs different location; 2 = "same finding"; 1 = vague; 0 = wrong.
EX027 — Property. State one property of allowed that must hold after any deny-list addition. Rubric: 3 = monotonic shrink / every deny host always denied; 2 = informal; 1 = wrong property; 0 = none.

Synthesis

Schemas, idempotency and properties make assessment software boring in the good sense: predictable under retry and review.

Continue studying — chapter 09 of 50

← Chapter 08 (draft) · Course chapter index · Chapter 10 (draft) →