Chapter 32 — Data governance: reliability
Lesson map
Lesson 32 published. Map, example, EX094.
flowchart TD
N0["Learning outcomes"]
N1["32.1 Why reconcile"]
N2["Lab — Reconciling 8,000 field reads with billing rec"]
N3["32.2 Reconciliation results"]
N4["32.3 Effect on leakage"]
N5["32.4 Tracking breaks"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX094"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["32.1 Why reconcile"]
N2["Lab — Reconciling 8,000 field reads with billing rec"]
N3["32.2 Reconciliation results"]
N4["32.3 Effect on leakage"]
N5["32.4 Tracking breaks"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX094"]
[Learning outcomes]
|
v
[32.1 Why reconcile]
|
v
[Lab — Reconciling 8,000 field reads with billing rec]
|
v
[32.2 Reconciliation results]
|
v
[32.3 Effect on leakage]
|
v
[32.4 Tracking breaks]
|
v
[Worked example] --> [Exercise EX094]
Learning outcomes
You will (1) reconcile data across systems; (2) quantify reconciliation gaps; (3) assign and track reconciliation breaks.
32.1 Why reconcile
The leakage figure subtracts billed volume from supplied volume. If billing misses meter readings, billed volume falls and leakage appears higher than it really is. Kofi reconciles field meter readings with readings recorded in billing to measure how reliably data pass from one system to the other. Reconciliation is one of the oldest data controls, used in accounting long before computers, and it remains effective because it compares independent records of the same facts rather than trusting either one. Kofi runs it monthly.
Lab — Reconciling 8,000 field reads with billing records
"""Reconcile monthly meter reads between field app and billing system (stdlib only)."""
import random
random.seed(27)
field = {f"M{i:05d}": round(random.uniform(10, 40), 1) for i in range(8000)}
billing = dict(field)
for k in random.sample(list(field), 120): del billing[k] # lost in transfer
for k in random.sample(list(billing), 60): billing[k] = round(billing[k] * 10, 1) # unit error
missing = [k for k in field if k not in billing]
mismatch = [k for k in billing if abs(billing[k] - field[k]) > 0.05]
print(f"field reads {len(field)}; missing in billing {len(missing)}; value mismatches {len(mismatch)}")
print(f"reconciled share {100*(len(field)-len(missing)-len(mismatch))/len(field):.2f}%")
Run with python3 lab_reconcile.py. Verified output:
field reads 8000; missing in billing 120; value mismatches 60
reconciled share 97.75%
32.2 Reconciliation results
The lab compares eight thousand synthetic field reads with billing. One hundred and twenty reads are missing from billing entirely, and sixty appear with a different value. The reconciled share, reads present with matching values, is 97.75 percent. Kofi explains that the two kinds of breaks have different causes and owners: missing reads usually come from transfer failures, while value mismatches usually come from manual corrections in billing that were not applied back to the meter data store. Each break type is counted separately.
32.3 Effect on leakage
Kofi estimates the effect of missing reads on the leakage figure. Because each missing read lowers billed volume, leakage is overstated. He calculates the volume associated with the missing reads in the synthetic data and shows that it shifts district leakage upward noticeably in the two districts with the most breaks. This explains why those districts appeared to be leaking more than neighbouring districts with similar pipes, and prevents repair crews being sent to the wrong places. The corrected figures are shared with planners.
32.4 Tracking breaks
Each break becomes a record with an owner, cause and status. Transfer failures go to the integration team; mismatches go to billing. Kofi sets a target to resolve breaks within ten working days and reports ageing weekly. Breaks older than the target are escalated to the governance board. Prof. Varga explains that reconciliation without break tracking measures problems without fixing them, and that ageing reports show whether teams are keeping up or falling behind. The first ageing report shows most breaks are closed within target.
Worked example
Kofi reconciles 8,000 field reads, finding 120 missing in billing and 60 mismatches for 97.75 percent reconciled, links breaks to overstated leakage and tracks breaks by owner and age.
Synthetic case. Reads are synthetic; outputs come from lab_reconcile.py.
Errors and limits
Reconciling a system with itself proves nothing; use independent records.
Exercises
- EX094 — Reconciled share. What share of reads reconciled? Rubric: 3 = 97.75%;; 2 = partly correct;; 1 = wrong;; 0 = none.
- EX095 — Missing reads effect. How do missing reads affect leakage? Rubric: 3 = they overstate it;; 2 = partly correct;; 1 = understate;; 0 = none.
- EX096 — Mismatch owner. Who owns value mismatches? Rubric: 3 = billing;; 2 = partly correct;; 1 = other;; 0 = none.
Continue studying — lesson 32 (chapter 32 of 50)
← Chapter 31 · Course chapter index · Chapter 33 →
Continue to the next authored lesson.