Chapter 4 — Scripting and data formats

Chapter 4 — Scripting and data formats

Lesson map

Lesson 04 published. Map, example, EX013.

flowchart TD
  N0["Learning outcomes"]
  N1["4.1 Small scripts"]
  N2["4.2 Parsing common formats"]
  N3["4.3 Validating data"]
  N4["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4
  N4 --> EX["Worked example"]
  EX --> Q["Exercise EX013"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["4.1 Small scripts"]
  N2["4.2 Parsing common formats"]
  N3["4.3 Validating data"]
  N4["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX013"]

[Learning outcomes]
    |
    v
[4.1 Small scripts]
    |
    v
[4.2 Parsing common formats]
    |
    v
[4.3 Validating data]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX013]

Learning outcomes

You will (1) write small scripts to read ledger data; (2) parse common formats; (3) validate data before use.

4.1 Small scripts

Aiko writes Python scripts with only the standard library to automate routine checks. Her first script asks a node for its latest block number and prints it with the time. A second compares the block numbers of all three nodes. She learns to structure scripts as small functions, add clear messages and handle errors such as a node not responding. Scripts make checks fast and consistent. Mr Kowal reviews her code and suggests storing settings like node addresses in one place, so they can be changed without editing every script.

4.2 Parsing common formats

Ledger tools return data in JSON, while the library's database exports CSV. Aiko uses Python's json and csv modules to read both. She extracts transfer events with sender, receiver, amount and block number, and loan records with borrower code, tool and dates. She learns that formats have quirks: numbers may arrive as text, and dates may use different formats. She converts each field explicitly. Parsing correctly is the foundation for every comparison later, because a misread field produces wrong results that look perfectly believable.

4.3 Validating data

Before using data, Aiko's scripts check it: amounts must be whole positive numbers, block numbers must increase, dates must be valid and borrower codes must exist in the database. Invalid records are listed separately rather than silently skipped. On a synthetic export she finds two loan records with return dates before borrow dates, caused by a data-entry error at the counter. Validation stops bad data from producing misleading results and often reveals process problems, like this one, that are worth fixing at the source rather than in every script.

Synthetic case. The scripts and data are synthetic.

Errors and limits

Silently skipping invalid records hides data problems that need fixing at the source.

Exercises

• EX013 — Settings advice. What did Mr Kowal suggest about settings? Rubric: 3 = store them in one place;; 2 = central settings;; 1 = hard-code everywhere;; 0 = none.
• EX014 — Format quirk. Name one format quirk Aiko meets. Rubric: 3 = numbers arriving as text;; 2 = date formats;; 1 = vague;; 0 = none.
• EX015 — Validation rule. Name one validation rule. Rubric: 3 = block numbers must increase;; 2 = amounts positive whole numbers;; 1 = anything goes;; 0 = none.
• EX016 — Invalid records found. What invalid records were found? Rubric: 3 = two loans returned before they were borrowed;; 2 = date errors;; 1 = vague;; 0 = none.

Synthesis

Small scripts that parse carefully and validate first make ledger data trustworthy.

Continue studying — chapter 04 of 18

← Chapter 03 (draft) · Course chapter index · Chapter 05 (draft) →