Chapter 26 — Automation and scheduling

Lesson map

Lesson 26 published. Map, example, EX076.

flowchart TD
  N0["Learning outcomes"]
  N1["26.1 Scheduling"]
  N2["26.2 Dependencies"]
  N3["26.3 Monitoring"]
  N4["26.4 Change control"]
  N0 --> N1 --> N2 --> N3 --> N4
  N4 --> EX["Worked example"]
  EX --> Q["Exercise EX076"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["26.1 Scheduling"]
  N2["26.2 Dependencies"]
  N3["26.3 Monitoring"]
  N4["26.4 Change control"]
  N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX076"]

[Learning outcomes]
    |
    v
[26.1 Scheduling]
    |
    v
[26.2 Dependencies]
    |
    v
[26.3 Monitoring]
    |
    v
[26.4 Change control]
    |
    v
[Worked example] --> [Exercise EX076]

Learning outcomes

You will (1) schedule data checks and loads; (2) handle dependencies and failures; (3) monitor automated jobs.

26.1 Scheduling

Ravi's checks, loads and reports now run on a schedule. Contractor files load at two in the morning, checks run at three, summary tables at four and dashboards refresh at five. He documents the schedule in a table with each job, its time, dependencies and owner. He notes that schedules make governance routine, but also create dependencies: if the load fails, every later job may run on incomplete data. The table is published with the dictionary so that users know when each report is refreshed each morning.

26.2 Dependencies

Ravi adds dependency checks: each job confirms that the previous job finished successfully and produced expected row counts before starting. If not, it stops and sends an alert. One night, a contractor file is late, and the load waits. The checks hold the summary job, and the dashboard shows an amber indicator rather than incomplete figures. He explains that controlled failure is better than silent success on bad data. The contractor's file arrives later that morning, and the jobs run in sequence once the load completes successfully.

26.3 Monitoring

Ravi creates a job monitoring summary each morning: which jobs ran, how long they took, records processed and any failures. Run times are tracked over time; when the check job slows down as data grow, he notices before it misses its window. He adds an index to the jobs table, which restores speed. Ms. Moreau notes that monitoring the governance machinery itself prevents gradual failures from going unnoticed until they cause problems. He records the change and the run times before and after it in the change log.

26.4 Change control

Changes to scheduled jobs follow a simple process: test on synthetic data, review by a colleague, record in the change log and deploy at a quiet time. When Ravi changes the export filter after the incident, he follows this process even under pressure. He notes that skipping change control during emergencies is tempting but often causes new problems, and that a quick but disciplined process serves better than shortcuts. The change log shows each change with its test result, reviewer and the time it was deployed.

Worked example

Ravi schedules loads, checks, summaries and dashboards with documented dependencies, stops jobs on failed predecessors, monitors run times and adds an index, and applies change control even during the incident.

Synthetic case. Jobs are synthetic.

Errors and limits

Jobs that run regardless of failed predecessors publish incomplete figures.

Exercises

• EX076 — Check time. When do checks run?
  Show marking criteria

  Rubric: 3 = three in the morning;; 2 = partly correct;; 1 = wrong;; 0 = none.
• EX077 — Slowdown fix. How did Ravi fix the slowing check job?
  Show marking criteria

  Rubric: 3 = added an index;; 2 = partly correct;; 1 = vague;; 0 = none.
• EX078 — Change steps. Name the change control steps.
  Show marking criteria

  Rubric: 3 = test, review, log and deploy at a quiet time;; 2 = most;; 1 = few;; 0 = none.