◇ OpenCyber Academy
Bachelor's · Chapter 6 · English

Chapter 6 — Python for data checks

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 6 — Python for data checks

Lesson map

Lesson 06 published. Map, example, EX016.

flowchart TD
  N0["Learning outcomes"]
  N1["6.1 Reading files"]
  N2["6.2 Writing checks"]
  N3["6.3 Reporting results"]
  N4["6.4 Repeatable work"]
  N0 --> N1 --> N2 --> N3 --> N4
  N4 --> EX["Worked example"]
  EX --> Q["Exercise EX016"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["6.1 Reading files"]
  N2["6.2 Writing checks"]
  N3["6.3 Reporting results"]
  N4["6.4 Repeatable work"]
  N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX016"]

[Learning outcomes]
    |
    v
[6.1 Reading files]
    |
    v
[6.2 Writing checks]
    |
    v
[6.3 Reporting results]
    |
    v
[6.4 Repeatable work]
    |
    v
[Worked example] --> [Exercise EX016]

Learning outcomes

You will (1) read data files with Python; (2) write simple checks; (3) report results clearly.

6.1 Reading files

Amara learns to read comma-separated files using Python's csv module. She opens a synthetic extract of loyalty sign-ups, reads each row as a dictionary keyed by column names and counts rows. She learns to specify the file encoding, because names with accents appeared garbled when she did not. She notes that reading data correctly is the first check: if the file is misread, every later result will be wrong, however careful the analysis appears. She adds the encoding to every script she writes afterwards.

6.2 Writing checks

She writes functions that test one rule each: postcode present, email containing an at sign, birth year between 1910 and 2010. Each function returns true or false for a row. A loop applies all functions and counts failures per rule. She keeps checks small and named clearly, so that a reader can see exactly what each one tests. Dr. Lindqvist explains that simple, readable checks are easier to trust and maintain than clever ones. Each function has a one-line comment describing the rule it implements and its source.

6.3 Reporting results

Amara's first report lists failures as raw counts. Her tutor asks what the counts mean without totals, so she adds percentages and the number of rows checked. She also lists a few example failing rows with personal details removed, so readers can see what problems look like. She learns that a good report answers what was checked, how much failed, why it matters and what should happen next, in that order. She uses this order in every later report, and her tutor marks the improvement.

6.4 Repeatable work

Amara saves her script with comments and the date of the extract used. When the tutor asks her to rerun it on a newer extract, it takes seconds. A classmate who checked data manually in a spreadsheet must repeat hours of work. She sees that scripts make checks repeatable and their logic visible, which matters for governance because checks should be run regularly and reviewed by others rather than performed once and forgotten. Her script is later shared with the Harbourline data team, who adopt it.

Worked example

Amara reads a sign-up extract with the right encoding, writes small named checks, reports counts with percentages and examples and reruns her script on a new extract in seconds.

Synthetic case. Extracts are synthetic.

Errors and limits

Reporting counts without totals hides the scale of problems.

Exercises

Continue studying — lesson 06 (chapter 06 of 40)

← Chapter 05 · Course chapter index · Chapter 07 →

Continue to the next authored lesson.

Worked answers

EX016. Names with accents appeared garbled.

EX017. Between 1910 and 2010.

EX018. What was checked, how much failed, why it matters, what next.