Chapter 12 — Operating systems
Lesson map
Lesson 12 published. Map, example, EX034.
flowchart TD
N0["Learning outcomes"]
N1["12.1 Files and permissions"]
N2["12.2 Scheduled jobs"]
N3["12.3 Audit logs"]
N4["12.4 Patching and change"]
N0 --> N1 --> N2 --> N3 --> N4
N4 --> EX["Worked example"]
EX --> Q["Exercise EX034"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["12.1 Files and permissions"]
N2["12.2 Scheduled jobs"]
N3["12.3 Audit logs"]
N4["12.4 Patching and change"]
N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX034"]
[Learning outcomes]
|
v
[12.1 Files and permissions]
|
v
[12.2 Scheduled jobs]
|
v
[12.3 Audit logs]
|
v
[12.4 Patching and change]
|
v
[Worked example] --> [Exercise EX034]
Learning outcomes
You will (1) relate operating system controls to data protection; (2) manage scheduled data jobs reliably; (3) keep audit logs useful.
12.1 Files and permissions
Tidewater's data processing runs on Linux servers. Kofi reviews file permissions on directories holding customer extracts and finds that one directory is readable by every user on the server. He explains that operating system permissions are the last line of defence when application controls fail. He changes the directory to be readable only by the processing account and the data team group, and documents the expected permissions so that later checks can compare actual settings with the intended ones. A weekly script compares them.
12.2 Scheduled jobs
Nightly jobs load meter readings, run quality rules and publish reports. Kofi finds that if a job fails, the next job runs anyway on incomplete data and publishes misleading results. He changes the schedule so each job checks that its predecessor finished successfully and produced the expected row counts. Jobs that cannot run send an alert. He notes that dependencies between jobs are data lineage in practice, and making them explicit prevents silent propagation of failures. The alert goes to the on-call data engineer.
12.3 Audit logs
The operating system records logins and commands. Kofi checks that logs from data servers are sent to a central log store, where administrators cannot alter them, and that they are kept for one year. He finds one server whose logs stayed local and were overwritten after a week. Without logs, Tidewater could not show who accessed a customer extract during an investigation. He adds the server to central logging and a weekly check that all data servers are reporting. The check emails missing servers.
12.4 Patching and change
Kofi joins the monthly patch cycle. Before patching data servers, the team runs a test job on a staging server to confirm that the new libraries do not change outputs. In one cycle, an updated date library changes how ambiguous timestamps are parsed. The test catches the difference before production. He learns that system maintenance can alter data processing silently, so governance needs regression tests for data outputs as well as for application features. The comparison script is kept and run on every patch cycle from then on.
Worked example
Kofi restricts a world-readable extract directory, chains jobs on successful predecessors, centralises a server's logs and catches a timestamp-parsing change in staging before patching.
Synthetic case. Servers are synthetic.
Errors and limits
Permissions drift after emergency fixes; check them regularly.
Exercises
- EX034 — Permission fix. What did Kofi change about the extract directory? Rubric: 3 = readable only by the processing account and data team;; 2 = partly correct;; 1 = vague;; 0 = none.
- EX035 — Job chaining. How are jobs now protected from bad inputs? Rubric: 3 = each checks its predecessor succeeded with expected counts;; 2 = mentions predecessor;; 1 = vague;; 0 = none.
- EX036 — Patch catch. What did the staging test catch? Rubric: 3 = a change in timestamp parsing;; 2 = partly correct;; 1 = vague;; 0 = none.
Continue studying — lesson 12 (chapter 12 of 50)
← Chapter 11 · Course chapter index · Chapter 13 →
Continue to the next authored lesson.