◇ OpenCyber Academy
Technologist · Chapter 6 · English

Chapter 6 — Data formats and exchange

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 6 — Data formats and exchange

Lesson map

Lesson 06 published. Map, example, EX016.

flowchart TD
  N0["Learning outcomes"]
  N1["6.1 Common formats"]
  N2["6.2 Exchange specifications"]
  N3["6.3 Validating files"]
  N4["6.4 Version changes"]
  N0 --> N1 --> N2 --> N3 --> N4
  N4 --> EX["Worked example"]
  EX --> Q["Exercise EX016"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["6.1 Common formats"]
  N2["6.2 Exchange specifications"]
  N3["6.3 Validating files"]
  N4["6.4 Version changes"]
  N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX016"]

[Learning outcomes]
    |
    v
[6.1 Common formats]
    |
    v
[6.2 Exchange specifications]
    |
    v
[6.3 Validating files]
    |
    v
[6.4 Version changes]
    |
    v
[Worked example] --> [Exercise EX016]

Learning outcomes

You will (1) compare common data formats; (2) define exchange specifications; (3) validate incoming files.

6.1 Common formats

Ravi compares formats used at Calder Valley: CSV for contractor exports, JSON from the tenant portal, fixed-width files from the old rent system and spreadsheets from managers. CSV is simple but has no types; JSON carries structure but can vary between records; fixed-width files need exact column positions. He uses Python's csv and json modules to read each, noting the specific problems each format tends to create. He writes a short note for each format describing its typical problems and how his scripts handle them, so that colleagues can reuse the approach.

6.2 Exchange specifications

To reduce problems, Ravi helps write an exchange specification for contractor files: file name pattern, encoding, delimiter, column names and order, data types, date format in year-month-day order, allowed values for status and priority and what to do when a job is cancelled. The repairs steward approves it and sends it to all contractors. He notes that a specification turns assumptions into agreements, making it possible to reject files that do not comply. The specification is published with an example file that contractors can test against.

6.3 Validating files

Ravi writes a validator that checks each incoming contractor file against the specification before loading it. Files passing all checks load automatically; files failing are held and the contractor is emailed a report listing problems. In the first month, two contractors send non-compliant files, both fixed within days. He notes that automated validation shifts the burden of fixing errors to the source, where they can be fixed properly. Contractors begin to test their files against the specification before sending them, and held files become rare after the first two months.

6.4 Version changes

When a contractor upgrades its system, a new column appears. The validator holds the file. Ravi checks with the steward, who agrees the column is harmless and updates the specification to version two, allowing an optional extra column. Ms. Moreau notes that specifications need version control like code, so everyone knows which version applies and changes are deliberate rather than accidental, with a record of who approved each change. The specification history is kept in the repository next to the validator code, so each version can be matched to the validator that enforced it.

Worked example

Ravi compares CSV, JSON, fixed-width and spreadsheet formats, helps write a contractor exchange specification, builds a validator that holds non-compliant files and manages a specification version change.

Synthetic case. Files and specifications are synthetic.

Errors and limits

Accepting files without validation moves source errors into reports.

Exercises

Continue studying — lesson 06 (chapter 06 of 30)

← Chapter 05 · Course chapter index · Chapter 07 →

Continue to the next authored lesson.