Chapter 4 — Scripting and data formats | Technical Authorized offensive security and red team

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 4 — Scripting and data formats

Lesson map

Lesson 04 published. Map, example, EX013.

flowchart TD
  N0["Learning outcomes"]
  N1["4.1 Why scripts"]
  N2["4.2 Untrusted data"]
  N3["4.3 Reproducibility"]
  N4["4.4 Formats"]
  N5["4.5 Secrets in scripts"]
  N6["4.6 Code review"]
  N7["4.8 Exit codes"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX013"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["4.1 Why scripts"]
  N2["4.2 Untrusted data"]
  N3["4.3 Reproducibility"]
  N4["4.4 Formats"]
  N5["4.5 Secrets in scripts"]
  N6["4.6 Code review"]
  N7["4.8 Exit codes"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX013"]

[Learning outcomes]
    |
    v
[4.1 Why scripts]
    |
    v
[4.2 Untrusted data]
    |
    v
[4.3 Reproducibility]
    |
    v
[4.4 Formats]
    |
    v
[4.5 Secrets in scripts]
    |
    v
[4.6 Code review]
    |
    v
[4.8 Exit codes]
    |
    v
[Worked example] --> [Exercise EX013]

Learning outcomes

You will (1) parse simple JSON/CSV evidence safely; (2) avoid executing untrusted artifacts; (3) keep scripts reproducible; (4) record interpreter versions.

4.1 Why scripts

Technicians automate inventory diffs, hash checks and ticket formatting, and scripts must be readable by the next person. Scripts exist so Sam can repeat lab_ports-style checks and hash evidence the same way every Saturday. They do not exist to invent new attack paths; every script for ENG-512 maps to a method card on the signed list. Script headers cite method card ids so automation cannot drift from that list, and orphan scripts without cites are deleted from the Saturday job folder.

4.2 Untrusted data

Files from targets are data, not code. Do not run unknown binaries or macros from a test target on an admin workstation. Untrusted data includes API JSON from staging and any file a tool writes. Sam validates shapes before parsing, and schema validation drops unknown JSON keys before logging. A surprise field that looks like a fare transaction id is redacted and escalated, not logged in full; payment-shaped fields from staging responses are treated as sensitive even in non-production, because staging data is sometimes copied from live systems.

4.3 Reproducibility

Pin Python versions, seed any randomness and keep input files beside the script in the evidence pack. Reproducibility means another technician can run the same script on the same inventory and get the same ok and NOT APPROVED lines. Sam pins script versions in the notebook beside the ENG-512 ticket id. Peers reproduce using the notebook's script hash, and mismatched hashes abort the comparison. Most reproducibility disputes turn out to be version disputes, and they end when the pins match.

4.4 Formats

JSON and CSV are common for tool exports, and Sam validates the schema before trusting fields for a finding. Formats for inventories and evidence indexes are CSV or plain text agreed in the runbook. Binary dumps and undocumented JSON blobs fail peer check because the next operator cannot read them under time pressure. Inventory CSV columns are name, ip, port, proto, approved. Reordering them breaks lab_ports, so Sam validates the header row before the job runs; a file with a missing or renamed column stops the job instead of being classified silently.

4.5 Secrets in scripts

No hard-coded passwords: scripts read from the vault or from an environment injected by the jump host. Those are the only patterns Sam accepts in code review for ENG-512 automation. Pre-merge secret scanning on the automation repository fails the build on tokens. Sam rotates any token that ever landed in a pull-request diff, because deleting the line does not delete the history that reviewers, mirrors and backups already hold. The rotation is recorded in the ENG-512 ticket with the commit that exposed the token, so a later reviewer can confirm the old value is dead.

4.6 Code review

Another technician reads Sam's script before it runs in an engagement window. Code review catches freestyle port lists and hard-coded passwords before a window. A peer score below pass means the script stays out of Saturday's job list. Reviewers grep for host patterns that look like fare subnets and reject those loops. Code review is a scope control, not only a style club. The reviewer's name and the script hash go into the ENG-512 notebook, so the job that runs on Saturday is provably the one that was read. A script changed after review returns to the queue, even for a one-line fix.

4.7 Input size limits

Scripts that read tool exports cap file size so a huge dump cannot exhaust the jump host. Caps are safety features, not pessimism. Input size limits stop a runaway inventory from locking the jump host. Sam caps rows processed per run and fails closed when the cap trips: the job prints ERROR: inventory exceeds cap and exits 2. Truncation without an error would hide 8080/http-alt past the cut line. When the cap trips, Sam checks the export with its owner before raising the cap in a reviewed change, rather than editing the limit on the jump host.

4.8 Exit codes

Sam's scripts exit non-zero on MISMATCH or NOT APPROVED so CI and humans notice failures instead of skimming green-looking logs. Exit codes distinguish ok listeners from NOT APPROVED findings so CI and humans agree; lab_ports-style jobs that always exit zero hide 3389/rdp until someone reads stdout carefully. The SOC monitors non-zero exits from ENG job users. Scripts that exit 0 while printing NOT APPROVED are bugs fixed before the next Saturday. Sam's review checklist includes running each job against a list that contains one unapproved port and confirming the non-zero exit.

4.9 Field discipline

Sam's scripting standard is short enough to remember: one purpose per script, argparse or explicit constants at the top, no network calls unless the ROE file path is passed in, and a --dry-run mode that prints what would happen. Dry-run saved him when a path typo would have pointed a hash job at a production share listing. Reviewers look for dry-run and for the absence of hard-coded hosts more than for clever code. Field discipline for scripting adds pinned versions, secret-free trees, size caps and exit codes that match the report, and Sam's ENG-512 notebook index links each script hash to the method card it implements.

Worked example (synthetic NMT)

Sam's hash-check script (Chapter 5 lab) takes a folder of evidence and prints MATCH/MISMATCH; it never opens images beyond hashing bytes.

Synthetic case — Northline Municipal Transit (NMT). The transit agency, Sam Park and all figures are invented for teaching.

Errors and limits

Live transit systems and fare equipment are never tested without a separate, specialised authorization. This book uses synthetic data only and grants no authority to test real systems.

Exercises

EX013 — Data not code. Why not double-click a .doc from the target web server? Rubric: 3 = may execute macros/malware on admin PC; treat as untrusted; 2 = partial; 1 = fine; 0 = none.
EX014 — Repro items. Name three things that make a script reproducible. Rubric: 3 = version, inputs, seed/settings; 2 = two; 1 = one; 0 = none.
EX015 — Secrets. How should a script get a staging password? Rubric: 3 = vault/env injection; not hard-coded; 2 = partial; 1 = hard-code; 0 = none.
EX016 — Peer read. Why does another technician read the script first? Rubric: 3 = catch scope/safety bugs; shared understanding; 2 = partial; 1 = bureaucracy; 0 = none.

Synthesis

Scripts are controlled tools: reproducible, secret-free and reviewed.

Continue studying — chapter 04 of 18

← Chapter 03 (draft) · Course chapter index · Chapter 05 (draft) →