## Chapter 14 — Software engineering ### Lesson map Lesson 14 published. Map, example, EX040. ```mermaid flowchart TD N0["Learning outcomes"] N1["14.1 Requirements for pipelines"] N2["14.2 Testing pipelines"] N3["14.3 Release management"] N4["14.4 Maintainability"] N0 --> N1 --> N2 --> N3 --> N4 N4 --> EX["Worked example"] EX --> Q["Exercise EX040"] ``` ### Logical drawing ```mermaid flowchart LR N0["Learning outcomes"] N1["14.1 Requirements for pipelines"] N2["14.2 Testing pipelines"] N3["14.3 Release management"] N4["14.4 Maintainability"] N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX040"] ``` ```text [Learning outcomes] | v [14.1 Requirements for pipelines] | v [14.2 Testing pipelines] | v [14.3 Release management] | v [14.4 Maintainability] | v [Worked example] --> [Exercise EX040] ``` ### Learning outcomes You will (1) apply requirements and testing practice to data pipelines; (2) manage releases of data logic; (3) design for maintainability. ### 14.1 Requirements for pipelines Kofi writes requirements for the meter data pipeline as testable statements: every accepted reading has a valid meter and unit; rejected readings are reported with reasons within one hour; daily totals for each district reconcile with the sum of readings. Each requirement has an owner and an acceptance test. He explains that vague requirements such as data should be accurate cannot be tested, so they cannot be shown to be met or traced when they fail. Owners sign off each acceptance test. ### 14.2 Testing pipelines Kofi builds three layers of tests. Unit tests check individual rules. Integration tests run a small synthetic dataset through the whole pipeline and compare outputs with expected results. Data tests run on production outputs and check properties such as row counts within expected ranges and no negative consumption. The data tests catch a problem unit tests could not: a vendor file arrives with a new column order, and readings land in the wrong fields until the reader is updated. A column-name check now runs first. ### 14.3 Release management Changes to pipeline logic follow a release process: code review, tests, a staging run on the previous month's synthetic replica and a comparison of outputs. Differences above agreed tolerances must be explained before release. Kofi argues that pipeline releases deserve the same care as customer-facing software, because a small change to a rule can change thousands of bills. Each release note links to the decision record that justified the rule change. Two releases were held back last quarter because unexplained differences exceeded the tolerances. ### 14.4 Maintainability Kofi documents the pipeline with a diagram, a rule catalogue and an operations guide. He removes unused code and replaces hard-coded thresholds with configuration values that data owners can review. Prof. Varga explains that maintainability is a governance concern: when the person who built a pipeline leaves, the organisation must still be able to explain and change it. Unmaintainable pipelines become black boxes whose outputs nobody can justify. A new team member uses the operations guide to rerun a failed load in her first week without help. ### Worked example Kofi writes testable pipeline requirements, builds unit, integration and data tests that catch a column-order change, adds release comparisons and documents the pipeline for maintainability. > **Synthetic case.** The pipeline is synthetic. ### Errors and limits Tolerances set too wide hide real changes; review them with owners. ### Exercises - **EX040 — Testable requirement.** Give one testable pipeline requirement. *Rubric:* 3 = accepted readings have valid meter and unit; rejects reported within an hour; totals reconcile;; 2 = one;; 1 = vague;; 0 = none. - **EX041 — Data test catch.** What did data tests catch? *Rubric:* 3 = a vendor file with a new column order;; 2 = partly correct;; 1 = vague;; 0 = none. - **EX042 — Hard-coded values.** What replaced hard-coded thresholds? *Rubric:* 3 = configuration values reviewed by owners;; 2 = partly correct;; 1 = vague;; 0 = none. ### Continue studying — lesson 14 (chapter 14 of 50) [← Chapter 13](#ch-13) · [Course chapter index](../SUBJECTS.md) · [Chapter 15 →](#ch-15) Continue to the next authored lesson. ## Worked answers **EX040.** Every accepted reading has a valid meter and unit. **EX041.** A vendor file arriving with a new column order. **EX042.** Configuration values that data owners can review.