Chapter 17 — Data architecture basics
Lesson map
Lesson 17 published. Map, example, EX049.
flowchart TD
N0["Learning outcomes"]
N1["17.1 Systems and flows"]
N2["17.2 Warehouses and lakes"]
N3["17.3 Lineage"]
N4["17.4 Architecture and governance"]
N0 --> N1 --> N2 --> N3 --> N4
N4 --> EX["Worked example"]
EX --> Q["Exercise EX049"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["17.1 Systems and flows"]
N2["17.2 Warehouses and lakes"]
N3["17.3 Lineage"]
N4["17.4 Architecture and governance"]
N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX049"]
[Learning outcomes]
|
v
[17.1 Systems and flows]
|
v
[17.2 Warehouses and lakes]
|
v
[17.3 Lineage]
|
v
[17.4 Architecture and governance]
|
v
[Worked example] --> [Exercise EX049]
Learning outcomes
You will (1) describe how data flow between systems; (2) explain warehouses and lakes; (3) connect architecture to governance.
17.1 Systems and flows
Amara maps Harbourline's architecture: tills, a store system, loyalty, products, finance, stock and a reporting warehouse. Data flow nightly from operational systems into the warehouse, where reports are built. Some flows are automated; others depend on staff exporting files. She notes that manual exports are where data most often go missing or arrive late, and marks them on her map as candidates for automation or at least for completeness checks. The map is reviewed with IT, who confirm two further manual exports she had missed.
17.2 Warehouses and lakes
A data warehouse holds cleaned, structured data organised for reporting. A data lake holds raw data in many formats for later analysis. Harbourline has a warehouse and is considering a lake for website click data. Amara explains that lakes without governance become swamps: nobody knows what files contain or whether they can be trusted. She recommends that any lake have catalogue entries, owners and retention rules from its first day. The lake project accepts these conditions before any click data are loaded into it.
17.3 Lineage
Lineage describes where data come from and how they change. Amara traces the board's weekly sales figure back through the warehouse to tills and the store system. She finds a transformation that excludes refunds, which nobody had documented. Finance assumed refunds were included. She records lineage in the catalogue entry for the figure. She explains that lineage helps people trust figures and find the cause when numbers look wrong. Finance and the reporting team agree to label the figure clearly as excluding refunds from then on.
17.4 Architecture and governance
Amara notes that architecture choices affect governance. Central warehouses make consistent definitions easier but can slow teams down. Distributed approaches give teams freedom but make consistency harder. Harbourline is small enough for a central warehouse with clear owners. Dr. Lindqvist explains that good architecture makes the right thing easy, such as applying classifications automatically, while poor architecture forces people to remember rules that systems could enforce for them. Amara recommends automatic classification in the warehouse as Harbourline's first architectural improvement for governance.
Worked example
Amara maps systems and flags manual exports, warns that ungoverned lakes become swamps, finds an undocumented refund exclusion through lineage and links architecture choices to governance.
Synthetic case. The architecture is synthetic.
Errors and limits
Lakes without catalogues and owners become unusable.
Exercises
- EX049 — Weak flows. Where do data most often go missing? Rubric: 3 = manual exports;; 2 = partly correct;; 1 = other;; 0 = none.
- EX050 — Refund finding. What did lineage reveal? Rubric: 3 = an undocumented exclusion of refunds;; 2 = partly correct;; 1 = vague;; 0 = none.
- EX051 — Lake recommendation. What should any lake have from day one? Rubric: 3 = catalogue entries, owners and retention rules;; 2 = all;; 1 = some;; 0 = none.
Continue studying — lesson 17 (chapter 17 of 40)
← Chapter 16 · Course chapter index · Chapter 18 →
Continue to the next authored lesson.