Chapter 14 — Automation
Lesson map
Lesson 14 published. Map, example, EX053.
flowchart TD
N0["Learning outcomes"]
N1["14.1 Automate"]
N2["14.2 Do not automate"]
N3["14.3 Windows"]
N4["14.4 Loop detection"]
N5["14.5 Output review"]
N6["14.6 Change control"]
N7["14.7 Idempotent jobs"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX053"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["14.1 Automate"]
N2["14.2 Do not automate"]
N3["14.3 Windows"]
N4["14.4 Loop detection"]
N5["14.5 Output review"]
N6["14.6 Change control"]
N7["14.7 Idempotent jobs"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX053"]
[Learning outcomes]
|
v
[14.1 Automate]
|
v
[14.2 Do not automate]
|
v
[14.3 Windows]
|
v
[14.4 Loop detection]
|
v
[14.5 Output review]
|
v
[14.6 Change control]
|
v
[14.7 Idempotent jobs]
|
v
[Worked example] --> [Exercise EX053]
Learning outcomes
You will (1) decide what to automate; (2) keep humans for validation; (3) schedule automation inside windows; (4) monitor automation for loops.
14.1 Automate
Hashing, port list diffs, ticket field checks, report skeletons.
The rule is to automate only the checks Sam already performs the same way every week, such as classifying inventories and hashing evidence. Jobs may repeat those proven paths, but they may not invent new methods, and automation that would introduce a new attack technique is refused for ENG-512. Method invention therefore stays off the automation backlog entirely; a new technique has to be written as a method card, reviewed and run by hand first.
14.2 Do not automate
Exploitation, scope decisions, severity without review.
Stop decisions and scope expansion also stay human. Operators halt on fare alerts, and jobs never decide on their own to continue. In job code, a fare-alert flag is a hard stop that freezes the run, not a soft warning that an operator may dismiss. Keeping these choices out of automation means a script cannot widen ENG-512 or keep testing through an alert simply because nobody was watching the console at that moment. Severity ratings likewise wait for a human reviewer before any ticket is raised.
14.3 Windows
Automation runs only inside the engagement window with the same rate limits.
Job schedules match the ROE Saturday window exactly. Overnight freestyle jobs against staging are change-control violations, however convenient they would be, and the scheduler access controls deny ENG users outside the signed window so the rule does not depend on memory. This keeps automated traffic inside the period when owners and on-call responders expect it and can tell a test from a real problem. When a job is still running as the window closes, it stops and the remainder waits for the next agreed slot.
14.4 Loop detection
Caps on retries and maximum runtime are mandatory.
Loop detection in each job kills the run when 401 responses repeat beyond the noise budget. The synthetic drill's 10:12 stop is the pattern automation must copy: the job halts, releases its connections and emits the same halt string the drill taught, so the engagement channel receives a message responders already recognise. A job without a retry cap and a runtime cap is not ready for the Saturday window, however well it performed in a quick test.
14.5 Output review
Every automated finding candidate gets human validation.
Output review is mandatory before any finding ships. Automated NOT APPROVED lines still need a human to check for DNS name mismatches, which the job cannot judge, so blind auto-filing is refused. Auto-ticket jobs stay disabled until that human mismatch gate returns approve. The review costs a few minutes per run, and it prevents owners from receiving tickets about hosts that were simply listed under a different name. Reviewers also confirm that each candidate maps to an approved host before it becomes a ticket.
14.6 Change control
Automation scripts are versioned like production code.
Change control on automation matches ROE change control. A new job cannot enter the Saturday window until it has passed peer review and the SOC has been notified, using the same template as an ROE change announcement. Adding a job silently is treated as a defect, because the SOC would see unexpected traffic with no notice explaining it and might respond as if it were a real attack. The job's version and its approving reviewer are recorded with the change notice.
14.7 Idempotent jobs
Automated diffs can run twice without doubling tickets. Sam keys tickets on host+port+date to avoid duplicates.
Idempotent design is what makes those re-runs safe. Sam's ENG-512 reporters upsert by listener key, so running a job a second time updates the existing ticket for that port instead of opening a duplicate. The listener-key upsert is covered by a unit test in the automation repo, so a later change to the reporter cannot quietly reintroduce duplicate tickets for owners to close.
14.8 Disable switch
Automation has a global disable flag the SOC can request. Sam tests that flag on day one of the engagement.
With that one flag, Sam can turn off all ENG automation if the journey planner misbehaves. The disable test is a Lab II item, and the switch must be exercised end to end in rehearsal; a switch that exists only on paper keeps the job off the Saturday list. Testing it early means that, if it is ever needed during a live window, nobody is discovering for the first time whether it actually works.
14.9 Field discipline
Automation pull requests need a safety block: rate limits, kill switch, dry-run, and which ROE clause allows the job. Without that block, reviewers reject the PR. Sam added the template after a well-meaning script once ran outside the window because cron used UTC and the author thought in local time.
Field discipline for automation is windowed, loop-aware, reviewed, disableable. Unattended cleverness is not a goal at NMT. A job that fails any one of those four properties stays off the Saturday list until the gap is fixed and reviewed again by a peer.
14.10 Closing note
Sam charts automation runtime per night. A job that suddenly takes ten times longer may be stuck in a loop or hitting a larger inventory — both deserve a human look before the next window.
Automation multiplies good method cards and also multiplies mistakes. Sam therefore automates a job only after a manual ENG-512 run has succeeded on the same path, with the method card, rate and evidence already reviewed. Automating an unproven path simply repeats an untested mistake on every scheduled run.
14.11 Extra
Caps include both retry count and wall-clock minutes. Sam's port-diff job dies after 20 minutes and pages him; a silent hang is treated like a failed stop switch.
As extra practice, Sam writes a disable drill into every automation README and actually runs it on the jump host before the job is scheduled. If the disable switch is untested, the job stays off the Saturday list, because a switch that has never been flipped gives no assurance at all. The README records when the drill was last run, so a reviewer can see that the switch was exercised recently.
Worked example (synthetic NMT)
Sam automates the port diff nightly on staging inventory exports during the engagement week; he still manually triages NOT APPROVED lines.
Synthetic case — Northline Municipal Transit (NMT). The transit agency, Sam Park and all figures are invented for teaching.
Errors and limits
Live transit systems and fare equipment are never tested without a separate, specialised authorization. This book uses synthetic data only and grants no authority to test real systems.
Exercises
- EX053 — Automate list. Name three good automation candidates. Rubric: 3 = three from chapter; 2 = two; 1 = one; 0 = none.
- EX054 — Never automate. Name two things Sam must not automate. Rubric: 3 = exploitation and scope decisions; 2 = one; 1 = hashing; 0 = none.
- EX055 — Retry cap. Why do automation retries need caps? Rubric: 3 = prevent lockouts/log storms/loops; 2 = partial; 1 = unlimited better; 0 = none.
- EX056 — Human gate. What always needs a human before a finding is filed? Rubric: 3 = validation of automated candidates; 2 = partial; 1 = vague; 0 = none.
Synthesis
Automate the repetitive; never automate permission, exploitation or unreviewed severity.
Continue studying — chapter 14 of 18
← Chapter 13 (draft) · Course chapter index · Chapter 15 (draft) →