Chapter 18 — Profiling data
Lesson map
Lesson 18 published. Map, example, EX052.
flowchart TD
N0["Learning outcomes"]
N1["18.1 What profiling is"]
N2["Lab — Completeness of five fields in 5,000 synthetic"]
N3["18.2 Results by field"]
N4["18.3 Results across rows"]
N5["18.4 From findings to actions"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX052"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["18.1 What profiling is"]
N2["Lab — Completeness of five fields in 5,000 synthetic"]
N3["18.2 Results by field"]
N4["18.3 Results across rows"]
N5["18.4 From findings to actions"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX052"]
[Learning outcomes]
|
v
[18.1 What profiling is]
|
v
[Lab — Completeness of five fields in 5,000 synthetic]
|
v
[18.2 Results by field]
|
v
[18.3 Results across rows]
|
v
[18.4 From findings to actions]
|
v
[Worked example] --> [Exercise EX052]
Learning outcomes
You will (1) profile a data set for completeness; (2) interpret field and row results; (3) turn findings into actions.
18.1 What profiling is
Profiling means examining data to understand their content and quality before making decisions. Amara profiles the synthetic loyalty member file, counting missing values for each field and how many records are complete on all fields. She explains that profiling is usually the first step in any quality work, because it shows where problems lie and how large they are, which helps owners choose priorities based on evidence rather than complaints. Amara explains each step of the profiling script to the loyalty steward, who will run it again later.
Lab — Completeness of five fields in 5,000 synthetic loyalty records
"""Profile completeness of five fields in 5,000 synthetic loyalty records (stdlib only)."""
import random
random.seed(7)
miss = {"member_id": 0.0, "postcode": 0.06, "email": 0.18, "birth_year": 0.31, "store_code": 0.02}
rows = [{f: (None if random.random() < p else "x") for f, p in miss.items()} for _ in range(5000)]
for f in miss:
print(f"{f:<11} complete {100*sum(r[f] is not None for r in rows)/len(rows):5.1f}%")
print(f"rows complete on all fields: {sum(all(v is not None for v in r.values()) for r in rows)}")
Run with python3 lab_profile.py. Verified output:
member_id complete 100.0%
postcode complete 93.9%
email complete 82.3%
birth_year complete 68.6%
store_code complete 98.3%
rows complete on all fields: 2594
18.2 Results by field
The lab profiles five thousand records. Member identifier is complete in every record. Store code is complete in 98.3 percent, postcode in 93.9 percent, email in 82.3 percent and birth year in only 68.6 percent. Amara notes that the pattern makes sense: identifiers are generated by the system, while email and birth year depend on members choosing to provide them, and birth year was optional on old sign-up forms. The new sign-up form makes birth year optional but asks for it with a short explanation of age-related offers.
18.3 Results across rows
Only 2,594 of the 5,000 records are complete on all five fields, about 51.9 percent. Amara explains that row completeness is lower than any single field because gaps fall on different records. If a campaign needs email, postcode and birth year together, it can reach only about half the members. She stresses that field-level figures alone would mislead marketing about how many members could receive a targeted offer. Marketing revises its campaign estimate accordingly and plans its targeting with the realistic reachable number of members.
18.4 From findings to actions
Amara discusses results with the loyalty owner. They agree that birth year matters only for age-related offers and need not be chased for everyone. Email matters for digital receipts and offers, so the till prompt will ask members without email. Postcode gaps come mainly from one old sign-up channel, which will be closed. She records each action with an owner and plans to repeat the profile in three months to measure change. The loyalty owner signs off the actions in the governance log.
Worked example
Amara profiles 5,000 records, finding member identifier complete, store code 98.3, postcode 93.9, email 82.3 and birth year 68.6 percent, with 2,594 rows complete on all fields, and agrees targeted actions.
Synthetic case. Records are synthetic; outputs come from lab_profile.py.
Errors and limits
Field completeness overstates usable rows; report row completeness too.
Exercises
- EX052 — Lowest field. Which field is least complete? Rubric: 3 = birth year at 68.6%;; 2 = partly correct;; 1 = other;; 0 = none.
- EX053 — Complete rows. How many records are complete on all fields? Rubric: 3 = 2,594;; 2 = partly correct;; 1 = wrong;; 0 = none.
- EX054 — Birth year action. What was decided about birth year? Rubric: 3 = not chased for everyone, needed only for age offers;; 2 = partly correct;; 1 = vague;; 0 = none.
Continue studying — lesson 18 (chapter 18 of 40)
← Chapter 17 · Course chapter index · Chapter 19 →
Continue to the next authored lesson.