Chapter 36 — Data governance: performance and scale
Lesson map
Lesson 36 published. Map, example, EX106.
flowchart TD
N0["Learning outcomes"]
N1["36.1 Growth"]
N2["36.2 Making controls efficient"]
N3["36.3 Control cost"]
N4["36.4 Value of controls"]
N0 --> N1 --> N2 --> N3 --> N4
N4 --> EX["Worked example"]
EX --> Q["Exercise EX106"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["36.1 Growth"]
N2["36.2 Making controls efficient"]
N3["36.3 Control cost"]
N4["36.4 Value of controls"]
N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX106"]
[Learning outcomes]
|
v
[36.1 Growth]
|
v
[36.2 Making controls efficient]
|
v
[36.3 Control cost]
|
v
[36.4 Value of controls]
|
v
[Worked example] --> [Exercise EX106]
Learning outcomes
You will (1) keep governance controls efficient as data grow; (2) measure control cost; (3) prioritise controls by value.
36.1 Growth
Smart meter readings grow from daily to hourly as Tidewater upgrades meters, multiplying volume by twenty-four. Acoustic loggers add more. Kofi estimates that the rule library would take far longer each night at the new volume if run unchanged. He explains that governance controls are workloads with costs, and that designs which work at today's scale may fail when volumes increase, causing late checks or skipped controls that undermine the whole governance approach. Kofi presents the estimate before the meter upgrade begins, so that changes are ready in time.
36.2 Making controls efficient
Kofi changes rules to process only new and changed records rather than full histories, partitions data by district so checks run in parallel and moves simple range checks into the ingestion step where data are already being read. He measures runtime on a synthetic dataset at the new volume and confirms the nightly checks now finish well before the morning deadline. He documents the design choices so later engineers understand why checks are structured this way. Range checks at ingestion also reject bad readings sooner.
36.3 Control cost
Kofi records the computing cost and staff time of each control. A detailed distribution comparison on every column costs far more than its findings justify for low-risk tables. He proposes risk-based frequency: high-risk data checked continuously, medium-risk daily and low-risk weekly. The governance board approves the approach. He notes that governance competes for resources with other work, so showing value for cost helps sustain support from managers who must fund it. Each data set's risk level is recorded in the catalogue, so the schedule follows automatically from the classification.
36.4 Value of controls
To prioritise, Kofi lists for each control the problems it detected over six months and their impact. Some controls found nothing, which could mean the risk is absent or the control is weak. He tests silent controls by injection before retiring any, and retires two that duplicate stronger checks. Prof. Varga stresses that controls which never fire should be tested rather than assumed useless, since silence can reflect success or failure. The remaining silent controls stay in place, because injection showed that each still detects the errors it was designed to catch.
Worked example
Kofi prepares controls for hourly meter data with incremental, partitioned and early checks, records control costs, applies risk-based frequency and tests silent controls before retiring duplicates.
Synthetic case. Volumes are synthetic.
Errors and limits
Retiring silent controls without testing them may remove needed protection.
Exercises
- EX106 — Volume growth. By how much does hourly reading multiply volume? Rubric: 3 = twenty-four times;; 2 = partly correct;; 1 = wrong;; 0 = none.
- EX107 — Efficiency change. Name one efficiency change. Rubric: 3 = incremental processing, district partitions or ingestion checks;; 2 = one;; 1 = vague;; 0 = none.
- EX108 — Silent controls. What did Kofi do before retiring silent controls? Rubric: 3 = tested them by injection;; 2 = partly correct;; 1 = vague;; 0 = none.
Continue studying — lesson 36 (chapter 36 of 50)
← Chapter 35 · Course chapter index · Chapter 37 →
Continue to the next authored lesson.