Chapter 31 — Data governance: implementation II

Lesson map

Lesson 31 published. Map, example, EX091.

flowchart TD
  N0["Learning outcomes"]
  N1["31.1 The catalogue"]
  N2["31.2 Automating metadata"]
  N3["31.3 Classification in practice"]
  N4["31.4 Visible governance"]
  N0 --> N1 --> N2 --> N3 --> N4
  N4 --> EX["Worked example"]
  EX --> Q["Exercise EX091"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["31.1 The catalogue"]
  N2["31.2 Automating metadata"]
  N3["31.3 Classification in practice"]
  N4["31.4 Visible governance"]
  N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX091"]

[Learning outcomes]
    |
    v
[31.1 The catalogue]
    |
    v
[31.2 Automating metadata]
    |
    v
[31.3 Classification in practice]
    |
    v
[31.4 Visible governance]
    |
    v
[Worked example] --> [Exercise EX091]

Learning outcomes

You will (1) implement metadata and catalogue entries for governed data; (2) automate ownership and classification; (3) make governance visible to users.

31.1 The catalogue

Kofi implements a data catalogue for the leakage chain. Each data set has an entry with its description, owner, steward, classification, quality rules, lineage links, refresh schedule and retention period. He populates entries for the seven upstream nodes and the board figure, then adds the source systems. Users searching for supplied volume now find one authoritative entry instead of several similar tables with unclear differences. He explains that a catalogue turns governance decisions into information people meet when they actually look for data.

31.2 Automating metadata

Manual catalogue entries go stale. Kofi automates what he can: schema details, refresh times and quality results are pulled from the warehouse and rule library each night. Owners and descriptions remain manual but are checked for staleness; entries not reviewed in a year are flagged to stewards. He notes that automation keeps technical facts accurate while human review keeps meaning accurate, and that the two need different maintenance processes because they change for different reasons and at different speeds. Stewards clear flagged entries monthly.

31.3 Classification in practice

Each data set is classified as public, internal, confidential or restricted. Customer billing volumes are confidential; the priority services register is restricted. Kofi links classifications to warehouse access policies, so changing a classification changes who can read the data without separate requests. When an analyst creates a derived table joining confidential billing data, the classification is inherited automatically. He tests inheritance with synthetic tables to confirm that derived data never receive a lower classification than their inputs. Access requests fall because permissions follow classification.

31.4 Visible governance

Kofi adds quality badges to catalogue entries showing the latest all-rules pass rate and service level performance. Users can see whether a data set is currently trustworthy before using it. Network engineers begin checking badges before planning repairs, and ask owners about poor scores. Prof. Varga observes that visibility creates healthy pressure, because owners prefer to fix problems rather than have colleagues see persistent poor scores, and users gain the information they need to judge suitability. Badges link to the rule descriptions behind each score.

Worked example

Kofi builds catalogue entries for the leakage chain, automates technical metadata with stale-entry checks, links classification to access with inheritance and adds quality badges that engineers consult.

Synthetic case. The catalogue is synthetic.

Errors and limits

Badges can mislead if rules are weak; link to rule descriptions.

Exercises

• EX091 — Automated fields. Which metadata are pulled automatically?
  Show marking criteria

  Rubric: 3 = schema details, refresh times and quality results;; 2 = most;; 1 = one;; 0 = none.
• EX092 — Inheritance rule. What classification does a derived table get?
  Show marking criteria

  Rubric: 3 = never lower than its inputs;; 2 = partly correct;; 1 = vague;; 0 = none.
• EX093 — Badge effect. How did badges change engineers' behaviour?
  Show marking criteria

  Rubric: 3 = they check badges and ask owners about poor scores;; 2 = partly correct;; 1 = vague;; 0 = none.