OpenCyber AcademyLesson map
Lesson 14 published. Map, example, EX040.
flowchart TD
N0["Learning outcomes"]
N1["14.1 Requirements for pipelines"]
N2["14.2 Testing pipelines"]
N3["14.3 Release management"]
N4["14.4 Maintainability"]
N0 --> N1 --> N2 --> N3 --> N4
N4 --> EX["Worked example"]
EX --> Q["Exercise EX040"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["14.1 Requirements for pipelines"]
N2["14.2 Testing pipelines"]
N3["14.3 Release management"]
N4["14.4 Maintainability"]
N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX040"]
[Learning outcomes]
|
v
[14.1 Requirements for pipelines]
|
v
[14.2 Testing pipelines]
|
v
[14.3 Release management]
|
v
[14.4 Maintainability]
|
v
[Worked example] --> [Exercise EX040]
Learning outcomes
You will (1) apply requirements and testing practice to data pipelines; (2) manage releases of data logic; (3) design for maintainability.
14.1 Requirements for pipelines
Kofi writes requirements for the meter data pipeline as testable statements: every accepted reading has a valid meter and unit; rejected readings are reported with reasons within one hour; daily totals for each district reconcile with the sum of readings. Each requirement has an owner and an acceptance test. He explains that vague requirements such as data should be accurate cannot be tested, so they cannot be shown to be met or traced when they fail. Owners sign off each acceptance test.
14.2 Testing pipelines
Kofi builds three layers of tests. Unit tests check individual rules. Integration tests run a small synthetic dataset through the whole pipeline and compare outputs with expected results. Data tests run on production outputs and check properties such as row counts within expected ranges and no negative consumption. The data tests catch a problem unit tests could not: a vendor file arrives with a new column order, and readings land in the wrong fields until the reader is updated. A column-name check now runs first.
14.3 Release management
Changes to pipeline logic follow a release process: code review, tests, a staging run on the previous month's synthetic replica and a comparison of outputs. Differences above agreed tolerances must be explained before release. Kofi argues that pipeline releases deserve the same care as customer-facing software, because a small change to a rule can change thousands of bills. Each release note links to the decision record that justified the rule change. Two releases were held back last quarter because unexplained differences exceeded the tolerances.
14.4 Maintainability
Kofi documents the pipeline with a diagram, a rule catalogue and an operations guide. He removes unused code and replaces hard-coded thresholds with configuration values that data owners can review. Prof. Varga explains that maintainability is a governance concern: when the person who built a pipeline leaves, the organisation must still be able to explain and change it. Unmaintainable pipelines become black boxes whose outputs nobody can justify. A new team member uses the operations guide to rerun a failed load in her first week without help.
Worked example
Kofi writes testable pipeline requirements, builds unit, integration and data tests that catch a column-order change, adds release comparisons and documents the pipeline for maintainability.
Synthetic case. The pipeline is synthetic.
Errors and limits
Tolerances set too wide hide real changes; review them with owners.
Exercises
- EX040 — Testable requirement. Give one testable pipeline requirement.
Show marking criteria
Rubric: 3 = accepted readings have valid meter and unit; rejects reported within an hour; totals reconcile;; 2 = one;; 1 = vague;; 0 = none. - EX041 — Data test catch. What did data tests catch?
Show marking criteria
Rubric: 3 = a vendor file with a new column order;; 2 = partly correct;; 1 = vague;; 0 = none. - EX042 — Hard-coded values. What replaced hard-coded thresholds?
Show marking criteria
Rubric: 3 = configuration values reviewed by owners;; 2 = partly correct;; 1 = vague;; 0 = none.