◇ OpenCyber Academy
Technical · Chapter 4 · English

Chapter 4 — Scripting and data formats

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 4 — Scripting and data formats

Lesson map

Lesson 04 published. Map, example, EX013.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["4.1 Formats you will meet"]
  N2["4.2 A validation order that fails closed"]
  N3["4.3 Record coordinates"]
  N4["4.5 Practical activity"]
  N5["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX013"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["4.1 Formats you will meet"]
  N2["4.2 A validation order that fails closed"]
  N3["4.3 Record coordinates"]
  N4["4.5 Practical activity"]
  N5["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX013"]

[Learning outcomes and connection]
    |
    v
[4.1 Formats you will meet]
    |
    v
[4.2 A validation order that fails closed]
    |
    v
[4.3 Record coordinates]
    |
    v
[4.5 Practical activity]
    |
    v
[Chapter synthesis]
    |
    v
[Worked example] --> [Exercise EX013]

Learning outcomes and connection

After this chapter you should be able to:

The technician's scripts in later chapters — decoders, auditors, detectors — all read data someone else produced. This chapter makes them strict about what they accept, because a script that silently "fixes" bad input produces confident wrong answers.

4.1 Formats you will meet

4.2 A validation order that fails closed

Check in an order where each step can assume the previous ones passed:

  1. Structure: the right number of fields. If not, nothing else in the record can be trusted — reject.
  2. Identity and type: known station and tag; unit as the schema says (a level in feet is not a level in metres); value parses as a number.
  3. Finiteness: float("nan") and float("inf") parse successfully but are not measurements — reject.
  4. Time: the timestamp parses in the agreed format; it is not in the future beyond a small tolerance.
  5. Plausibility: value within the physical range of the instrument and process.

Reject what cannot be interpreted (wrong structure, wrong unit, not a number). Quarantine what is interpretable but implausible (out of range, future time): a real sensor fault or a real attack both look like this, and an operator must see it rather than have it silently dropped. Never coerce: turning "12,5" into 12.5 or 125 by guessing the decimal separator creates data that was never measured.

4.3 Record coordinates

Every finding should say where it came from — file, line, and field — so that someone else can verify it. csv.reader.line_num counts physical lines read so far, including the header, so the first data row is line 2. Quoted fields containing newlines make physical lines differ from records; line_num reports the line where the record ended, which is still a usable coordinate.

When you paste a finding into a ticket, keep the line number next to the station name, so the next person can open the export and go straight to the row. If a quoted field spans lines, say so, because the coordinate then marks the end of the record rather than its start.

4.4 Worked example: validating a station export (synthetic)

An export from Kestrel Water's historian lists one reading per station. The analysis time is 06:00Z on 12 October 2026.

Lab 4 (save as lab04.py, run with python3 lab04.py):

# Lab 4 - validate a telemetry export with record coordinates (offline, standard library only)
# Synthetic CSV export from Kestrel Water stations; "now" is 2026-10-12T06:00:00Z.
import csv, io, json, math
from datetime import datetime, timezone
export = """station,tag,value,unit,ts
KW-01,LEVEL_M,3.42,m,2026-10-12T05:58:00Z
KW-02,CL2_MGL,0.61,mg/L,2026-10-12T05:58:00Z
KW-03,LEVEL_M,,m,2026-10-12T05:58:00Z
KW-04,CL2_MGL,7.9,mg/L,2026-10-12T05:58:00Z
KW-05,FLOW_LS,"12,5",L/s,2026-10-12T05:58:00Z
KW-06,LEVEL_M,2.10,ft,2026-10-12T05:58:00Z
KW-07,LEVEL_M,2.95,m,2026-10-12 05:58
KW-08,LEVEL_M,nan,m,2026-10-12T05:58:00Z
KW-09,FLOW_LS,41.0,L/s,2026-10-12T07:15:00Z
KW-10,LEVEL_M,1.88,m,2026-10-12T05:58:00Z,extra
"""
SCHEMA = {"LEVEL_M": ("m", 0.0, 12.0), "CL2_MGL": ("mg/L", 0.0, 5.0), "FLOW_LS": ("L/s", 0.0, 400.0)}
now = datetime(2026, 10, 12, 6, 0, tzinfo=timezone.utc)
result = {"accept": [], "reject": [], "quarantine": []}
reader = csv.reader(io.StringIO(export))
header = next(reader)
for row in reader:
    where = f"line {reader.line_num}"
    if len(row) != len(header): result["reject"].append((where, f"{len(row)} fields, expected {len(header)}")); continue
    r = dict(zip(header, row))
    unit, lo, hi = SCHEMA[r["tag"]]
    if r["unit"] != unit: result["reject"].append((where, f"{r['station']} unit {r['unit']!r}, expected {unit!r}")); continue
    try: v = float(r["value"])
    except ValueError: result["reject"].append((where, f"{r['station']} value {r['value']!r} not a number")); continue
    if not math.isfinite(v): result["reject"].append((where, f"{r['station']} value not finite")); continue
    try:
        ts = datetime.strptime(r["ts"], "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=timezone.utc)
    except ValueError: result["reject"].append((where, f"{r['station']} timestamp {r['ts']!r} not RFC 3339 UTC")); continue
    if ts > now: result["quarantine"].append((where, f"{r['station']} timestamp {(ts - now).seconds // 60} min in the future")); continue
    if not lo <= v <= hi: result["quarantine"].append((where, f"{r['station']} {r['tag']}={v} outside {lo}-{hi}")); continue
    result["accept"].append((where, r["station"]))
for k in ("reject", "quarantine"):
    for where, why in result[k]: print(f"{k:<10} {where:<8} {why}")
print(json.dumps({k: len(v) for k, v in result.items()}))

Expected output (Python 3.13.5):

reject     line 4   KW-03 value '' not a number
reject     line 6   KW-05 value '12,5' not a number
reject     line 7   KW-06 unit 'ft', expected 'm'
reject     line 8   KW-07 timestamp '2026-10-12 05:58' not RFC 3339 UTC
reject     line 9   KW-08 value not finite
reject     line 11  6 fields, expected 5
quarantine line 5   KW-04 CL2_MGL=7.9 outside 0.0-5.0
quarantine line 10  KW-09 timestamp 75 min in the future
{"accept": 2, "reject": 6, "quarantine": 2}

Only two of ten rows are accepted. Each rejection names its line: an empty value (line 4), a decimal comma inside a quoted field (line 6), a level in feet (line 7), a timestamp without seconds or offset (line 8), nan (line 9) and an extra field (line 11). The quarantined rows are the interesting ones: chlorine at 7.9 mg/L is outside the instrument range — a sensor fault, a scaling error or a manipulated value — and KW-09's timestamp is 75 minutes in the future, which in Chapter 7 you will learn to suspect as a clock problem. Both are passed to a person with their coordinates.

The JSON summary on the last line is deliberately machine-readable, so that a monitoring job can alert when the rejection count rises.

4.5 Practical activity

Add a station-identity check to Lab 4: only KW-01 to KW-12 are valid. Then add a row KW-13,LEVEL_M,2.0,m,2026-10-12T05:58:00Z and confirm it is rejected with its line number. Finally, make the timestamp parser accept an explicit +00:00 offset as well as Z without accepting naive timestamps.

Reject KW-13 with its line number and a reason that names the allowed set. For timestamps, accept both Z and an explicit plus-zero offset, and still reject a bare local time. Re-run the lab and confirm that the JSON summary's reject count rises by one for the new station.

4.6 Common errors and limits

Exercises — chapter 4

EX013 — Predict before running. Objective: reason about the validator. Inputs: Lab 4 with three new rows: KW-11,LEVEL_M,12.0,m,2026-10-12T05:59:00Z; KW-12,LEVEL_M,inf,m,2026-10-12T05:59:00Z; KW-02,CL2_MGL,-0.0,mg/L,2026-10-12T05:59:00Z. Steps: predict each verdict and the new summary, then run. Expected evidence: predictions, actual output, explanation of any surprise. Rubric: 3 = accept, reject (not finite), accept; summary {"accept": 4, "reject": 7, "quarantine": 2}; notes −0.0 should be normalized; 2 = one wrong but explained; 1 = guesses; 0 = none.

EX014 — Reject or quarantine? Objective: apply the distinction. Inputs: (a) flow −3.0 L/s; (b) unit "M" instead of "m"; (c) level 11.9 m on a 12 m tank; (d) timestamp 4 s in the future. Steps: decide and justify. Expected evidence: four verdicts. Rubric: 3 = (a) quarantine (interpretable, implausible; might be reverse flow or offset); (b) reject or normalize only if the schema explicitly allows case-insensitive units — not guess; (c) accept, perhaps with a high-level alarm; (d) accept within an agreed clock tolerance; 2 = one weak; 1 = all reject; 0 = none.

EX015 — JSON with traps. Objective: harden JSON parsing. Inputs: {"station": "KW-03", "value": NaN}. Steps: show what json.loads returns by default and write the one-line change that rejects it. Expected evidence: code and behaviour. Rubric: 3 = default returns nan float; json.loads(s, parse_constant=lambda c: (_ for _ in ()).throw(ValueError(c))) or an equivalent function that raises; explains RFC 8259 does not allow NaN; 2 = works without explanation; 1 = attempted; 0 = none.

EX016 — Summary for monitoring. Objective: design a machine-readable result. Inputs: Lab 4. Steps: extend the JSON summary with counts by reason and by station, and propose one alert condition on it. Expected evidence: example JSON and the condition. Rubric: 3 = stable keys, counts by reason, alert such as "any quarantine of CL2_MGL, or rejects > 10 % of rows for two consecutive exports"; 2 = no reason breakdown; 1 = prose only; 0 = none.

Chapter synthesis

Operational data arrive malformed, ambiguous and occasionally hostile. Parse with real parsers, validate in an order that fails closed, never coerce, and report every finding with coordinates. Reject what you cannot interpret; quarantine what you can interpret but should not believe; and keep summaries machine-readable so that the validator itself can be monitored.

Continue studying — chapter 04 of 18

← Chapter 03 · Course chapter index · Chapter 05 →

Continue to the next authored lesson.

Worked answers

EX013 — Predict before running. KW-11 12.0 → accept (inclusive range); KW-12 inf → reject "value not finite" (line 13); KW-02 −0.0 → accept. Summary {"accept": 4, "reject": 7, "quarantine": 2} (verified by running). Normalize −0.0 to 0.0 before storage.

EX014 — Reject or quarantine? (a) Quarantine: interpretable but outside range — reverse flow, sensor offset or manipulation. (b) Reject unless the schema explicitly defines units case-insensitively; never guess. (c) Accept; a high-level process alarm is the operator's tool, not the validator's. (d) Accept if within an agreed clock tolerance (for example 5 s); otherwise quarantine as a clock issue.

EX015 — JSON with traps. json.loads('{"station": "KW-03", "value": NaN}') returns {'station': 'KW-03', 'value': nan}. Define def no_const(c): raise ValueError(f"non-standard JSON constant {c}") and call json.loads(s, parse_constant=no_const); it now raises. RFC 8259 does not permit NaN or Infinity.

EX016 — Summary for monitoring. Example: {"accept": 2, "reject": 6, "quarantine": 2, "by_reason": {"not a number": 2, "unit": 1, "timestamp": 1, "not finite": 1, "field count": 1, "range": 1, "future": 1}, "by_station": {"KW-03": 1, ...}}. Alert: any quarantine of a CL2_MGL tag, or rejects above 10 % of rows in two consecutive exports, or the export missing entirely.