Chapter 14 — Application and software supply chain security: automation
Lesson map
Lesson 14 published. Map, example, EX053.
flowchart TD
N0["Learning outcomes"]
N1["14.1 The nightly job"]
N2["14.2 Routing"]
N3["14.3 Permissions"]
N4["14.4 Silent failures"]
N5["14.5 Reviewing the job"]
N6["14.6 Time saved"]
N7["Building in steps"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX053"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["14.1 The nightly job"]
N2["14.2 Routing"]
N3["14.3 Permissions"]
N4["14.4 Silent failures"]
N5["14.5 Reviewing the job"]
N6["14.6 Time saved"]
N7["Building in steps"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX053"]
[Learning outcomes]
|
v
[14.1 The nightly job]
|
v
[14.2 Routing]
|
v
[14.3 Permissions]
|
v
[14.4 Silent failures]
|
v
[14.5 Reviewing the job]
|
v
[14.6 Time saved]
|
v
[Building in steps]
|
v
[Worked example] --> [Exercise EX053]
Learning outcomes
You will (1) automate inventory, advisory comparison and baseline checks; (2) route results to tickets; (3) limit automation's permissions; (4) watch for silent failures.
14.1 The nightly job
Each night a job builds the plugin inventory, compares it with advisories, checks headers and permissions for every site, and writes a report. Combining the checks into one job means every site is checked the same way every night. The job reuses the logic from the course labs: the inventory, the version comparison, the header baseline and the permission listing. Its report replaces the manual weekly checks that previously took a full day. The report lists coverage first, then new findings by severity.
14.2 Routing
Each finding becomes a ticket for the support queue with severity and due date. Highs also notify the duty supervisor. Turning findings into tickets uses the queue support already watches, so nothing needs a separate dashboard. Due dates follow the planning rule of high within 7 days and medium within 30. Notifying the duty supervisor for highs ensures an advisory like the resize-plugin one is not left waiting until morning. Each ticket links back to the nightly report line that produced it.
14.3 Permissions
The job reads site data and opens tickets; it cannot change sites. Updates are applied by technicians through the normal process. A read-only job limits the harm a bug or a stolen credential can do. The expired read credential in the worked example shows the job does depend on credentials, which is another reason they grant only reading. Updates still follow the procedure, with backups, hash checks and customer notice. The job's credential is also scoped to reading site data only.
14.4 Silent failures
If the report is missing or covers fewer sites than expected, an alert fires. A job that silently checks nothing is worse than no job, because it looks like good news. The coverage alert caught exactly this: zero findings for a cluster, together with 60 fewer sites than expected. Comparing sites checked against sites expected turns that silence into a visible alert. The alert goes to the duty supervisor, who checks the job before trusting the next report. The expected site count comes from Pinecrest's hosting records, not from the job itself.
14.5 Reviewing the job
The job's code is reviewed and tested with synthetic sites before each change. Testing on synthetic sites first means a change cannot break the real nightly report unnoticed. When the job's scripts first ran beside the manual checks, two differences were found, both bugs in the scripts. The same comparison approach is used whenever the job's checks change, with a second technician reviewing. A second technician signs off each change, and the review is recorded with the job's code in version control.
14.6 Time saved
Weekly manual checks that took a day now take an hour of reading tickets. The saved day goes into fixing findings rather than finding them. Grace reads only the summary each morning, and technicians work from the tickets, so the job's output is read by the people who act on it. The comparison is rough, since the old manual checks covered fewer sites, but the direction is clear. Tomás records the estimate in his project evidence as an approximate figure rather than a measured result.
Building in steps
The job started as Tomás's manual checks turned into scripts one at a time. Each script ran beside the manual check for two weeks, and results were compared before the manual step stopped. Two differences were found, both bugs in the scripts, and fixed before anyone relied on them. The report starts with counts by severity and coverage (sites checked versus sites expected), then lists new findings. Coverage at the top makes gaps hard to miss. Grace reads only the summary each morning; technicians work from the tickets.
Change control
Any change to the job's checks is reviewed by a second technician before it runs nightly.
Worked example
In its third week, the nightly job reports 0 findings for a whole hosting cluster. Because the site count also dropped by 60, the missing-coverage alert fires; the cause is an expired read credential for that cluster, which is renewed the same morning.
Synthetic case. Jobs and events are synthetic.
Errors and limits
Automation that can change sites needs far stronger review; keep it read-only where possible.
Exercises
- EX053 — Job tasks. What does the nightly job do? Rubric: 3 = inventory, advisory compare, headers and permissions, report; 2 = three; 1 = one; 0 = none.
- EX054 — Job permissions. Why can't the job change sites? Rubric: 3 = limits harm; changes by technicians; 2 = partial; 1 = vague; 0 = none.
- EX055 — Zero findings. Why was zero findings suspicious? Rubric: 3 = site count dropped too; 2 = partial; 1 = vague; 0 = none.
- EX056 — Cause. What caused the gap? Rubric: 3 = expired read credential; 2 = partial; 1 = vague; 0 = none.
Synthesis
Automate checks read-only and watch coverage.
Continue studying — chapter 14 of 18
← Chapter 13 (draft) · Course chapter index · Chapter 15 (draft) →