Chapter 19 — Data quality rules
Lesson map
Lesson 19 published. Map, example, EX055.
flowchart TD
N0["Learning outcomes"]
N1["19.1 Rules from standards"]
N2["19.2 Automating checks"]
N3["19.3 Thresholds and escalation"]
N4["19.4 Maintaining rules"]
N0 --> N1 --> N2 --> N3 --> N4
N4 --> EX["Worked example"]
EX --> Q["Exercise EX055"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["19.1 Rules from standards"]
N2["19.2 Automating checks"]
N3["19.3 Thresholds and escalation"]
N4["19.4 Maintaining rules"]
N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX055"]
[Learning outcomes]
|
v
[19.1 Rules from standards]
|
v
[19.2 Automating checks]
|
v
[19.3 Thresholds and escalation]
|
v
[19.4 Maintaining rules]
|
v
[Worked example] --> [Exercise EX055]
Learning outcomes
You will (1) write quality rules from standards; (2) automate checks; (3) set thresholds and escalation.
19.1 Rules from standards
Amara turns the loyalty data standard into rules. Each rule has an identifier, a description, the field it checks, the condition, the owner and the threshold. For example, rule LQ-03 checks that email addresses contain one at sign and a domain, owned by the loyalty steward, with a target of 99 percent validity among records with an email. Writing rules this way makes them testable, repeatable and traceable back to the standard approved by the governance group. The rule catalogue is kept with the standard.
19.2 Automating checks
She implements rules in a Python script that runs each night after the loyalty load. Results go to a small results table with date, rule, records checked and records failing. A simple chart shows trends. Amara explains that automated checks catch problems soon after they appear, while manual checks happen too rarely and depend on someone remembering. The script logs its own runs, so a missing night is noticed rather than silently skipped. The steward receives a short email summary each morning with the results.
19.3 Thresholds and escalation
Each rule has a threshold. If results fall below it, the steward receives an alert. If they stay below for a week, the owner is informed. If they stay below for a month, the governance group reviews the issue. Amara notes that escalation turns measurements into action, and that clear steps prevent both ignoring problems and overreacting to single bad nights caused by temporary system issues outside anyone's control. Amara documents the escalation steps in the standard so that every steward follows the same process.
19.4 Maintaining rules
Rules need maintenance. When Harbourline opens a store in a new region with a different postcode format, the postcode rule wrongly rejects valid codes. Amara updates the rule after the steward reports the false alerts, records the change and reruns past results. She learns that rules must change as the business changes, and that every rule change should be reviewed and documented like any other change to an important system. The change log shows who approved the update, when it was applied and why it was needed.
Worked example
Amara writes rules with identifiers, owners and thresholds, automates nightly checks with logged runs, defines three-step escalation and updates the postcode rule for a new region.
Synthetic case. Rules are synthetic.
Errors and limits
Rules that never change eventually reject valid data.
Exercises
- EX055 — Rule LQ-03. What does rule LQ-03 check? Rubric: 3 = emails contain one at sign and a domain;; 2 = partly correct;; 1 = vague;; 0 = none.
- EX056 — Month-long breach. Who reviews issues below threshold for a month? Rubric: 3 = the governance group;; 2 = partly correct;; 1 = other;; 0 = none.
- EX057 — Postcode change. Why did the postcode rule change? Rubric: 3 = a new region used a different format;; 2 = partly correct;; 1 = vague;; 0 = none.
Continue studying — lesson 19 (chapter 19 of 40)
← Chapter 18 · Course chapter index · Chapter 20 →
Continue to the next authored lesson.