Chapter 4 — Scripting and data formats
Lesson map
Lesson 04 published. Map, example, EX010.
flowchart TD
N0["Learning outcomes"]
N1["4.1 Formats"]
N2["4.2 Practical activity — Lab J"]
N3["4.3 Unknown is not no"]
N4["4.4 Rerunnable scripts"]
N5["4.6 Hands-on activity"]
N6["4.8 A small CSV example"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX010"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["4.1 Formats"]
N2["4.2 Practical activity — Lab J"]
N3["4.3 Unknown is not no"]
N4["4.4 Rerunnable scripts"]
N5["4.6 Hands-on activity"]
N6["4.8 A small CSV example"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX010"]
[Learning outcomes]
|
v
[4.1 Formats]
|
v
[4.2 Practical activity — Lab J]
|
v
[4.3 Unknown is not no]
|
v
[4.4 Rerunnable scripts]
|
v
[4.6 Hands-on activity]
|
v
[4.8 A small CSV example]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX010]
Learning outcomes
You will (1) read and write JSON and CSV in Python; (2) summarise inventory exports; (3) treat missing values as unknown; and (4) write scripts others can rerun.
4.1 Formats
Inventories arrive as JSON from APIs and CSV from spreadsheets and scanners. Python's json and csv modules read both. Always check the fields you need are present and have expected types.
At HHG, scanner exports come as CSV and the endpoint inventory as JSON. Check the record count and field names before trusting any summary. If a field is missing or has the wrong type, stop and report it rather than guessing. A scanner CSV with fewer rows than the endpoint JSON lists hosts usually means a site was missed, not that hosts vanished.
4.2 Practical activity — Lab J
# Lab J (ch 4): parse a synthetic JSON export of TLS endpoints and summarise key exchange by property.
import json, collections
export = json.loads("""[
{"property": "harbour", "host": "pms-01", "group": "X25519MLKEM768"},
{"property": "harbour", "host": "pos-02", "group": "secp256r1"},
{"property": "summit", "host": "pms-01", "group": "X25519"},
{"property": "summit", "host": "lock-hub", "group": null},
{"property": "lakeside","host": "pms-01", "group": "X25519MLKEM768"}]""")
summary = collections.defaultdict(collections.Counter)
for e in export:
g = e["group"]
kind = "unknown" if g is None else ("hybrid" if "MLKEM" in g else "classical")
summary[e["property"]][kind] += 1
for prop, c in sorted(summary.items()):
print(f"{prop:9s} hybrid={c['hybrid']} classical={c['classical']} unknown={c['unknown']}")
Expected output:
harbour hybrid=1 classical=1 unknown=0
lakeside hybrid=1 classical=0 unknown=0
summit hybrid=0 classical=1 unknown=1
The Summit lock hub has no recorded group. The script counts it as unknown, not classical, because nobody has observed it yet.
The script reads a JSON export, decides for each endpoint whether its group is hybrid, classical or unknown, and counts results per property. Note that null becomes unknown, not classical, and that hybrid is detected by the MLKEM part of the name. Lakeside's PMS negotiates hybrid while Summit's does not, which gives a first property-level comparison. Run it, then add an endpoint of your own and check the counts change as expected.
4.3 Unknown is not "no"
Treating missing values as a negative answer makes reports look complete and hides work. An unknown means "find out", and should create a task: ask the vendor, check configuration or observe traffic.
At HHG, the Summit lock hub stays unknown until the lock vendor answers or traffic is observed. Count unknowns separately in every report so readers can see how much is still to be found. Give each unknown an owner and a date, so it is closed rather than carried forward.
4.4 Rerunnable scripts
Put the input path and output format in arguments, print what was read (number of records), and keep scripts in version control with a short README. A script someone else can rerun next month is worth far more than a one-off calculation.
Write the script so it fails loudly on bad input rather than producing a wrong summary. Include an example input and its expected output, as the labs do. Print errors clearly, naming the file and row that caused them.
4.5 Worked example
An operator receives endpoints.csv from a scanner. Their script counts hybrid, classical and unknown per property, prints the total rows read (48), and writes a small CSV for the dashboard. When the scanner output adds a new column next month, the script still works because it reads columns by name.
The row count lets a reader check that nothing was dropped: if the scanner produced 48 rows, the summary must account for 48. Reading by name rather than position is what made the new column harmless.
4.6 Hands-on activity
Modify Lab J to also print the share of hybrid endpoints across all properties.
Compute the share as hybrid endpoints divided by all endpoints, and also as hybrid divided by endpoints with a known group, and print both with labels. With Lab J's data, the answers differ because of the Summit lock hub. Explain in one sentence which figure belongs in a report and why. Keep the unknown count visible next to both figures, and test the script with an empty export to make sure it does not divide by zero.
4.7 Common errors and limits
- Reading columns by position.
- Silently skipping rows with errors.
- Not printing how many records were read.
Each error produces silent mistakes. Reading by position breaks as soon as a scanner adds a column, as happened in the worked example. Skipping bad rows makes totals look complete when they are not, and the skipped rows may be the interesting ones. Without a printed record count, nobody can check whether the script read the whole file. Add a test for each to your scripts.
4.8 A small CSV example
The same idea works for CSV. Read with csv.DictReader so each row is a dictionary keyed by column name; check that the column group exists before the loop; and for each row decide hybrid, classical or unknown with the same rule as Lab J. Print the counts and the number of rows read, and write any row that failed validation to a separate file for review rather than discarding it. Rejected rows are often the most interesting ones — a scanner that could not complete a handshake may be pointing at exactly the firewall problem Chapter 13 teaches you to diagnose.
When you hand the script to a colleague, include an example input file with a few synthetic rows and the output it should produce. That small test lets them check the script still works after they change it, the same way the labs in this book show their expected output.
Keep the validation rule in one place in the script.
Exercises
- EX010 — Share. What share of Lab J's endpoints is hybrid? Rubric: 3 = 2/5 = 40 %.
- EX011 — Validation. Name three checks a script should do on each inventory record. Rubric: 3 = required fields, type/allowed values, duplicates.
- EX012 — Unknown task. Write the task you would raise for the Summit lock hub. Rubric: 3 = what to find, how, owner.
Synthesis
Simple, validated, rerunnable scripts turn inventory exports into honest summaries in which unknowns become tasks.
Continue studying — chapter 04 of 18
← Chapter 03 (draft) · Course chapter index · Chapter 05 (draft) →