Chapter 28 — Data governance: implementation I
Lesson map
Lesson 28 published. Map, example, EX082.
flowchart TD
N0["Learning outcomes"]
N1["28.1 The rule library"]
N2["Lab — Pass rates for three meter reading rules on 10"]
N3["28.2 Running the rules"]
N4["28.3 Interpreting failures"]
N5["28.4 Against thresholds"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX082"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["28.1 The rule library"]
N2["Lab — Pass rates for three meter reading rules on 10"]
N3["28.2 Running the rules"]
N4["28.3 Interpreting failures"]
N5["28.4 Against thresholds"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX082"]
[Learning outcomes]
|
v
[28.1 The rule library]
|
v
[Lab — Pass rates for three meter reading rules on 10]
|
v
[28.2 Running the rules]
|
v
[28.3 Interpreting failures]
|
v
[28.4 Against thresholds]
|
v
[Worked example] --> [Exercise EX082]
Learning outcomes
You will (1) implement data quality rules in code; (2) measure pass rates; (3) interpret combined results.
28.1 The rule library
Kofi implements the first quality rules in a shared library used by every pipeline. Each rule is a function with an identifier, description, owner and threshold read from configuration. The first three rules check that a meter identifier is present, that a reading is not below the previous reading for the same meter and that daily use is under 300 cubic metres, an implausible amount for any domestic or small business customer in Tidewater's area. Every rule is also documented in the catalogue.
Lab — Pass rates for three meter reading rules on 10,000 synthetic rows
"""Run validation rules on synthetic meter readings and report pass rates (stdlib only)."""
import random
random.seed(8)
rows = []
for i in range(10000):
prev = random.uniform(100, 5000); cur = prev + random.uniform(0, 60)
if random.random() < 0.02: cur = prev - random.uniform(1, 50) # rollback
if random.random() < 0.01: cur = prev + random.uniform(500, 2000) # spike
mid = f"M{i:06d}" if random.random() > 0.005 else ""
rows.append((mid, prev, cur))
rules = {
"meter id present": lambda r: bool(r[0]),
"reading not below previous": lambda r: r[2] >= r[1],
"daily use under 300 m3": lambda r: r[2] - r[1] < 300,
}
for name, f in rules.items():
ok = sum(1 for r in rows if f(r)); print(f"{name:28s} pass {ok/len(rows):.3f}")
allok = sum(1 for r in rows if all(f(r) for f in rules.values()))
print(f"rows passing all rules {allok} of {len(rows)} ({100*allok/len(rows):.1f}%)")
Run with python3 lab_rules.py. Verified output:
meter id present pass 0.995
reading not below previous pass 0.980
daily use under 300 m3 pass 0.990
rows passing all rules 9650 of 10000 (96.5%)
28.2 Running the rules
The lab runs the rules on 10,000 synthetic readings. The meter identifier rule passes 99.5 percent of rows, the not-below-previous rule 98.0 percent and the daily use rule 99.0 percent. Rows passing all three rules number 9,650, or 96.5 percent. Kofi notes that the combined rate is lower than any single rate, because failures fall on different rows. Owners see their own rule rates, but the combined figure is what billing actually receives. Kofi therefore asks the dashboard to show the all-rules figure prominently, with the individual rule rates shown underneath for owners who need to investigate their own failures.
28.3 Interpreting failures
Kofi examines failures. Readings below the previous value come mainly from meter replacements, where the new meter starts at zero. That is not an error in the reading but a missing link between old and new meters. He works with metering to record replacements so the rule can compare readings from the same meter only. Missing identifiers come from manual reads keyed without scanning, which the field application will now prevent by requiring a scan. Both fixes are scheduled for the next release.
28.4 Against thresholds
Compared with the thresholds agreed in chapter 24, validity at 96.5 percent falls short of the 98 percent target. Kofi reports this honestly to the governance board with the causes found and the fixes planned, estimating that the replacement fix alone would raise the combined pass rate above target. Prof. Varga notes that reporting a miss with a credible plan builds more trust than adjusting thresholds downward to make figures look acceptable. The board accepts the plan and asks for an update after the next release, with the rates recalculated on fresh data.
Worked example
Kofi implements a shared rule library, runs three rules on 10,000 rows with pass rates of 99.5, 98.0 and 99.0 percent and 96.5 percent passing all, traces causes and reports the miss against target.
Synthetic case. Data are synthetic; outputs come from lab_rules.py.
Errors and limits
Averaging rule rates overstates quality; report the all-rules rate.
Exercises
- EX082 — All-rules rate. What share of rows passed all rules? Rubric: 3 = 96.5% (9,650 of 10,000);; 2 = partly correct;; 1 = partial;; 0 = none.
- EX083 — Below-previous cause. What mainly caused readings below the previous value? Rubric: 3 = meter replacements starting at zero;; 2 = partly correct;; 1 = vague;; 0 = none.
- EX084 — Miss handling. How did Kofi handle missing the target? Rubric: 3 = reported it honestly with causes and fixes;; 2 = partly correct;; 1 = vague;; 0 = none.
Continue studying — lesson 28 (chapter 28 of 50)
← Chapter 27 · Course chapter index · Chapter 29 →
Continue to the next authored lesson.