Chapter 13 — Lineage for reports

Chapter 13 — Lineage for reports

Lesson map

Lesson 13 published. Map, example, EX037.

flowchart TD
  N0["Learning outcomes"]
  N1["13.1 Tracing a figure"]
  N2["13.2 Transformations"]
  N3["13.3 Investigating discrepancies"]
  N4["13.4 Keeping lineage"]
  N0 --> N1 --> N2 --> N3 --> N4
  N4 --> EX["Worked example"]
  EX --> Q["Exercise EX037"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["13.1 Tracing a figure"]
  N2["13.2 Transformations"]
  N3["13.3 Investigating discrepancies"]
  N4["13.4 Keeping lineage"]
  N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX037"]

[Learning outcomes]
    |
    v
[13.1 Tracing a figure]
    |
    v
[13.2 Transformations]
    |
    v
[13.3 Investigating discrepancies]
    |
    v
[13.4 Keeping lineage]
    |
    v
[Worked example] --> [Exercise EX037]

Learning outcomes

You will (1) trace report figures back to sources; (2) document transformations; (3) use lineage to investigate discrepancies.

13.1 Tracing a figure

The monthly report shows the percentage of emergency repairs completed within twenty-four hours. Ravi traces it: the report reads a summary table, which is built by a script from the jobs table, which receives contractor updates from the portal and call handler entries. He records each step, including filters and calculations, in a lineage note. He finds that the script excludes cancelled jobs, which is correct, but also excludes jobs without a completion time, which is not documented. He reports this to the steward.

13.2 Transformations

Ravi documents each transformation: which records are included, how times are calculated and what happens to missing values. He finds that the calculation uses the date only, not the time, so a job raised at eleven at night and completed at nine the next morning appears as one day, while a job raised at one in the morning and completed at eleven at night the same day appears as zero days. He recommends using full timestamps. The steward agrees, and the script is changed after testing.

13.3 Investigating discrepancies

The repairs manager's own spreadsheet shows a different emergency figure from the report. Using lineage, Ravi compares steps and finds that the spreadsheet includes jobs without completion times as late, while the report excludes them. Neither approach is documented. The steward decides that jobs without completion times should be reported separately as unknown. Both the report and spreadsheet are updated to match, ending months of disagreement. Ravi records the agreed approach in the lineage note, and the repairs manager retires his spreadsheet in favour of the corrected report a month later.

13.4 Keeping lineage

Ravi stores lineage notes next to each report in the repository and adds a rule that changes to report scripts must update the notes. He also adds a short lineage summary to each report's footer. Ms. Moreau explains that lineage helps people trust figures and resolve disagreements quickly, and that technologists are well placed to maintain it because they understand both the scripts and the business questions behind them. Report readers begin to consult the footer when figures surprise them, which reduces the number of questions sent to the data team.

Worked example

Ravi traces the emergency repair figure, finds undocumented exclusion of missing completion times and date-only calculations, resolves a disagreement with the manager's spreadsheet and stores lineage notes with each report.

Synthetic case. Reports are synthetic.

Errors and limits

Date-only calculations distort time-based targets.

Exercises

• EX037 — Undocumented exclusion. What was excluded without documentation? Rubric: 3 = jobs without a completion time;; 2 = partly correct;; 1 = vague;; 0 = none.
• EX038 — Timestamp issue. Why should full timestamps be used? Rubric: 3 = date-only calculations misstate elapsed time;; 2 = partly correct;; 1 = vague;; 0 = none.
• EX039 — Unknown handling. How are jobs without completion times reported now? Rubric: 3 = separately as unknown;; 2 = partly correct;; 1 = late;; 0 = none.

Continue studying — lesson 13 (chapter 13 of 30)

← Chapter 12 · Course chapter index · Chapter 14 →

Continue to the next authored lesson.