Chapter 22 — Data governance: system and data models
Lesson map
Lesson 22 published. Map, example, EX064.
flowchart TD
N0["Learning outcomes"]
N1["22.1 Modelling the chain"]
N2["Lab — Upstream lineage and ownership gaps for the le"]
N3["22.2 What the trace shows"]
N4["22.3 Closing the gaps"]
N5["22.4 Keeping models current"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX064"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["22.1 Modelling the chain"]
N2["Lab — Upstream lineage and ownership gaps for the le"]
N3["22.2 What the trace shows"]
N4["22.3 Closing the gaps"]
N5["22.4 Keeping models current"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX064"]
[Learning outcomes]
|
v
[22.1 Modelling the chain]
|
v
[Lab — Upstream lineage and ownership gaps for the le]
|
v
[22.2 What the trace shows]
|
v
[22.3 Closing the gaps]
|
v
[22.4 Keeping models current]
|
v
[Worked example] --> [Exercise EX064]
Learning outcomes
You will (1) model data flows and lineage for an indicator; (2) find ownership gaps from a lineage graph; (3) keep models current.
22.1 Modelling the chain
Kofi models the leakage indicator as a directed graph. The board report figure depends on a key performance indicator, which depends on billed volume and supplied volume. Billed volume depends on meter readings and tariffs; meter readings depend in part on manual readings from a field application; supplied volume depends on raw flow sensor data. Each node is a data set or calculation with an owner. The graph makes dependencies explicit, which a written description of the process tends to blur. Nodes are named.
Lab — Upstream lineage and ownership gaps for the leakage figure
"""Trace upstream lineage of a published figure through a synthetic pipeline graph (stdlib only)."""
edges = {
"board_report.leakage_pct": ["kpi.leakage"],
"kpi.leakage": ["billing.volume_billed", "scada.volume_supplied"],
"billing.volume_billed": ["meters.reading", "customers.tariff"],
"scada.volume_supplied": ["sensors.flow_raw"],
"meters.reading": ["field_app.manual_read"],
"customers.tariff": [], "sensors.flow_raw": [], "field_app.manual_read": [],
}
owners = {"kpi.leakage": "ops analytics", "billing.volume_billed": "billing", "scada.volume_supplied": "network ops",
"meters.reading": "metering", "customers.tariff": "billing", "sensors.flow_raw": None, "field_app.manual_read": None}
def up(n, seen):
for p in edges.get(n, []):
if p not in seen: seen.add(p); up(p, seen)
return seen
src = up("board_report.leakage_pct", set())
print(f"upstream nodes {len(src)}; sources {sorted(n for n in src if not edges[n])}")
print(f"nodes without an owner: {sorted(n for n in src if owners.get(n) is None)}")
Run with python3 lab_lineage.py. Verified output:
upstream nodes 7; sources ['customers.tariff', 'field_app.manual_read', 'sensors.flow_raw']
nodes without an owner: ['field_app.manual_read', 'sensors.flow_raw']
22.2 What the trace shows
Running the lab traces seven upstream nodes behind the board figure. Three are sources with no further inputs: customer tariffs, manual readings from the field application and raw flow sensor data. Two nodes have no owner: the manual readings and the raw sensor flows. Both feed the figure the regulator sees. Kofi notes that the most basic inputs, closest to the physical world, are exactly where ownership is missing, because they sit between engineering teams and nobody claimed them. Kofi reports both gaps.
22.3 Closing the gaps
Kofi proposes owners for the two gaps: the metering manager for manual readings and the network operations manager for raw flow data. Each accepts after agreeing what ownership means: approving quality rules, responding to issues within agreed times and signing off changes. He adds owners to the graph and reruns the trace, which now shows no unowned nodes. The rerun becomes a monthly check, since new data sets added to the chain could reopen gaps. Both owners sign a short ownership statement.
22.4 Keeping models current
Lineage models age as systems change. Kofi proposes that any change to the leakage pipeline must update the graph in the same release, enforced by a review checklist. He also compares the graph with the warehouse's query logs every quarter to find dependencies that exist in practice but not in the model. Prof. Varga explains that a model which drifts from reality is worse than none, because people trust it and make decisions based on incorrect assumptions. Two such dependencies appear. Both are added promptly.
Worked example
Kofi models the leakage chain as a graph, traces seven upstream nodes with three sources and two unowned nodes, assigns owners and keeps the model current with release checks and log comparisons.
Synthetic case. The graph is synthetic; outputs come from lab_lineage.py.
Errors and limits
Static lineage misses runtime dependencies; compare with logs.
Exercises
- EX064 — Upstream count. How many upstream nodes are behind the board figure? Rubric: 3 = seven;; 2 = partly correct;; 1 = wrong;; 0 = none.
- EX065 — Unowned nodes. Which nodes had no owner? Rubric: 3 = field_app.manual_read and sensors.flow_raw;; 2 = both;; 1 = one;; 0 = none.
- EX066 — Drift check. How does Kofi find dependencies missing from the model? Rubric: 3 = quarterly comparison with query logs;; 2 = partly correct;; 1 = vague;; 0 = none.
Continue studying — lesson 22 (chapter 22 of 50)
← Chapter 21 · Course chapter index · Chapter 23 →
Continue to the next authored lesson.