Chapter 4 — Scripting and data formats

Lesson map

Lesson 04 published. Map, example, EX013.

flowchart TD
  N0["Learning outcomes"]
  N1["4.1 Summarising reports"]
  N2["Lab — Weekly report summary from CSV"]
  N3["4.2 JSON configuration"]
  N4["4.3 Avoiding personal data"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX013"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["4.1 Summarising reports"]
  N2["Lab — Weekly report summary from CSV"]
  N3["4.2 JSON configuration"]
  N4["4.3 Avoiding personal data"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX013"]

[Learning outcomes]
    |
    v
[4.1 Summarising reports]
    |
    v
[Lab — Weekly report summary from CSV]
    |
    v
[4.2 JSON configuration]
    |
    v
[4.3 Avoiding personal data]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX013]

Learning outcomes

You will (1) write a script to summarise reports from a CSV file; (2) use JSON for configuration; (3) avoid storing personal data unnecessarily.

4.1 Summarising reports

Once a reporting mailbox exists, Priya writes a Python script that reads a weekly CSV export with date, depot, category and verdict, and prints counts by depot and category. She uses Python's csv module so commas inside fields are handled correctly. She tests the script on a small file whose totals she knows. The summary replaces a manual count that took the IT lead an hour each week and often contained mistakes, freeing time for replying to reporters. The IT lead checks the first output.

Lab — Weekly report summary from CSV

# Weekly report summary from CSV (synthetic).
import csv, io, json
cfg = json.loads('{"depots": ["North", "South", "East"]}')
data = "date,depot,category,verdict\n" + "".join(f"2026-10-0{1 + i % 5},{d},{c},{v}\n" for i, (d, c, v) in enumerate(
    [("North", "fake carrier", "malicious")] * 7 + [("North", "newsletter", "harmless")] * 5 +
    [("South", "fake invoice", "malicious")] * 4 + [("South", "other", "harmless")] * 3 + [("East", "fake carrier", "malicious")] * 5))
counts = {d: 0 for d in cfg["depots"]}
for row in csv.DictReader(io.StringIO(data)): counts[row["depot"]] += 1
print(counts, "total", sum(counts.values()))

Run with python3 lab_weekly.py. Verified output:

{'North': 12, 'South': 7, 'East': 5} total 24

4.2 JSON configuration

Priya stores settings, such as depot names and category labels, in a JSON file rather than inside the script. JSON is a simple text format of names and values that both people and programs can read. When a new depot opens, the IT lead updates the JSON file without changing code. Separating configuration from code makes scripts easier to maintain and reduces the chance of errors when someone less familiar with programming needs to make a change. She adds a short example file.

4.3 Avoiding personal data

The first export Priya receives includes reporter email addresses. She asks for them to be removed, because the summary needs only depot-level counts, and names could make staff feel monitored. The export now includes a random code instead. She stores files in a restricted folder and deletes them after four weeks. Good scripting practice includes thinking about what data the script handles and whether it needs them at all. The IT lead agrees with the change and records it in the data processing notes, and Tom praises her for raising the question early.

Worked example

Priya writes a CSV summary script, moves settings to JSON and removes reporter addresses from the export.

Synthetic case. Data are synthetic.

Errors and limits

Exports can change format. Random codes can sometimes be linked.

Exercises

• EX013 — CSV module reason. Why use Python's csv module?
  Show marking criteria

  Rubric: 3 = it handles commas inside fields;; 2 = parsing;; 1 = split;; 0 = none.
• EX014 — Lab top depot. Which depot had most reports in the lab?
  Show marking criteria

  Rubric: 3 = North with 12;; 2 = North;; 1 = vague;; 0 = none.
• EX015 — JSON benefit. Why store settings in JSON?
  Show marking criteria

  Rubric: 3 = changes need no code edits;; 2 = maintainability;; 1 = vague;; 0 = none.
• EX016 — Removed field. Which field was removed from the export?
  Show marking criteria

  Rubric: 3 = reporter email addresses;; 2 = addresses;; 1 = vague;; 0 = none.

Synthesis

Scripts should be tested, configurable and careful with personal data.

Continue studying — chapter 04 of 18

← Chapter 03 (draft) · Course chapter index · Chapter 05 (draft) →