◇ OpenCyber Academy
Technologist · Chapter 3 · English

Chapter 3 — Programming foundations

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 3 — Programming foundations

Lesson map

Lesson 03 published. Map, example, EX007.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["3.1 Untrusted input"]
  N2["3.2 Layers of validation"]
  N3["3.3 Reject, quarantine, accept"]
  N4["3.4 Small-code discipline"]
  N5["3.6 Practical activity: validator and test cases"]
  N6["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX007"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["3.1 Untrusted input"]
  N2["3.2 Layers of validation"]
  N3["3.3 Reject, quarantine, accept"]
  N4["3.4 Small-code discipline"]
  N5["3.6 Practical activity: validator and test cases"]
  N6["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX007"]

[Learning outcomes and connection]
    |
    v
[3.1 Untrusted input]
    |
    v
[3.2 Layers of validation]
    |
    v
[3.3 Reject, quarantine, accept]
    |
    v
[3.4 Small-code discipline]
    |
    v
[3.6 Practical activity: validator and test cases]
    |
    v
[Chapter synthesis]
    |
    v
[Worked example] --> [Exercise EX007]

Learning outcomes and connection

After this chapter you should be able to:

Implementers in critical services write a great deal of small code: parsers, glue between systems, checks, report generators. Most of it is never reviewed. This chapter is about writing that code so that it fails safely, because in LANTERN the code that reads station messages decides what the historian — and therefore the operators — believe about the water network.

3.1 Untrusted input

Untrusted input is any data whose content you do not control: messages from field devices, files from suppliers, fields typed by users, records from another organization's system. Even a device you own becomes untrusted once you remember that it may be misconfigured, faulty, replaced or impersonated. The rule is simple: check input against what you expect before acting on it, and decide in advance what happens when it fails the check. The OWASP project's guidance on input validation recommends allowlisting (accepting only known-good forms) over denylisting (blocking known-bad forms), validating both syntax and meaning, and doing it as early as possible (OWASP Input Validation Cheat Sheet).

3.2 Layers of validation

For a telemetry message of the form STATION|TAG|VALUE|UNIX_TIME|SEQ, validation has layers, and the order matters:

  1. Structure — exactly five fields. A wrong field count means the message cannot be interpreted at all.
  2. Identity and vocabulary — the station identifier matches the allowed pattern exactly (PS- and two digits) and the tag is one the system knows. Unknown tags are rejected, not stored "just in case".
  3. Type — value, time and sequence parse as numbers. Values that parse but are not finite (nan, inf) are rejected; many languages happily parse them.
  4. Physical meaning — the value lies within what the physical process can produce (a reservoir level between 0 and 12 metres; chlorine between 0 and 5 mg/L in this synthetic design). A value outside that range is not necessarily malicious — sensors fail — so it is quarantined for a person to review rather than silently dropped or silently stored.
  5. Time — the message's timestamp is close to the receiver's clock. A large difference means a clock problem or an old message; quarantine it.
  6. Order — the sequence number is greater than the last accepted for that station. A repeated or lower number is a replay or a reordering; reject it.

3.3 Reject, quarantine, accept

A validation function that returns only "valid" or "invalid" forces a bad choice for borderline data: either throw away a reading an operator might need, or store one that might mislead. Operational systems need a third outcome. Reject means the input cannot be trusted or interpreted and is logged and discarded. Quarantine means the input is interpretable but implausible; it is kept separately, flagged and reviewed. Accept means it passed every check and may be acted on. Each outcome is counted: a sudden rise in rejects is itself a monitoring signal (Chapter 19).

3.4 Small-code discipline

Four habits make small tools trustworthy. Keep one function per decision, so it can be tested alone. Make the state explicit — the last-seen sequence per station is passed in, not hidden in a global. Fail closed — any path you did not anticipate ends in reject, not accept. And test with bad cases first: for every check, write at least one input that should fail it, and run them all after every change.

Lab 3 follows all four at small scale: every decision sits in one short validate() function that can be tested alone. The sequence check receives the last accepted number per station as an argument, which is why the repeated 102 can be tested on its own, and the chlorine plausibility rule is a separate step driven by the LIMITS table after the value is parsed, which is why 9.80 mg/L can be quarantined rather than rejected without touching the parser.

3.5 Worked example: the station message validator (synthetic)

Lab 3 implements the six layers. Running it over eleven synthetic messages produces three accepts, six rejects and two quarantines. Look at the cases that a naive parser would get wrong: nan parses as a float and would be stored; 210.5;DROP is caught only because the value is parsed as a number rather than passed through as text; PS-7 fails because the pattern is matched in full; a repeated sequence number 102 is rejected as a replay even though the message is otherwise perfect; and a chlorine value of 9.80 mg/L is quarantined, not rejected, because a person needs to know whether the analyser at PS-07 has failed.

3.6 Practical activity: validator and test cases

Lab 3 (save as lab03.py, run with python3 lab03.py):

# Lab 3 - validating telemetry messages before they reach the historian (offline, standard library only)
# Synthetic message format from Fallowmere pumping stations:  STATION|TAG|VALUE|UNIX_TIME|SEQ
import re
LIMITS = {"LEVEL_M": (0.0, 12.0), "FLOW_LS": (0.0, 450.0), "CL2_MGL": (0.0, 5.0)}
STATION = re.compile(r"PS-\d{2}")
def validate(line, now, last_seq):
    parts = line.strip().split("|")
    if len(parts) != 5:
        return "reject: field count"
    st, tag, val, ts, seq = parts
    if not STATION.fullmatch(st):
        return "reject: station id"
    if tag not in LIMITS:
        return "reject: unknown tag"
    try:
        v, t, q = float(val), int(ts), int(seq)
    except ValueError:
        return "reject: not numeric"
    if v != v or v in (float("inf"), float("-inf")):
        return "reject: not finite"
    lo, hi = LIMITS[tag]
    if not lo <= v <= hi:
        return "quarantine: out of physical range"
    if abs(now - t) > 300:
        return "quarantine: clock skew >300 s"
    if q <= last_seq.get(st, -1):
        return "reject: replay or reorder"
    last_seq[st] = q
    return "accept"
now = 1_790_000_000
msgs = ["PS-04|LEVEL_M|3.42|1789999990|101", "PS-04|LEVEL_M|3.44|1789999995|102",
        "PS-04|LEVEL_M|3.44|1789999995|102", "PS-07|CL2_MGL|0.61|1789999801|55",
        "PS-07|CL2_MGL|9.80|1789999990|56", "PS-07|FLOW_LS|nan|1789999990|57",
        "PS-7|FLOW_LS|120|1789999990|58", "PS-09|PRESS_BAR|2.1|1789999990|9",
        "PS-09|FLOW_LS|210.5|1789990000|10", "PS-09|FLOW_LS|210.5;DROP|1789999990|11",
        "PS-04|LEVEL_M|3.47|1789999999"]
state = {}
counts = {}
for m in msgs:
    r = validate(m, now, state)
    counts[r.split(":")[0]] = counts.get(r.split(":")[0], 0) + 1
    print(f"{m:<38} {r}")
print(counts)

Expected output (Python 3.13.5):

PS-04|LEVEL_M|3.42|1789999990|101      accept
PS-04|LEVEL_M|3.44|1789999995|102      accept
PS-04|LEVEL_M|3.44|1789999995|102      reject: replay or reorder
PS-07|CL2_MGL|0.61|1789999801|55       accept
PS-07|CL2_MGL|9.80|1789999990|56       quarantine: out of physical range
PS-07|FLOW_LS|nan|1789999990|57        reject: not finite
PS-7|FLOW_LS|120|1789999990|58         reject: station id
PS-09|PRESS_BAR|2.1|1789999990|9       reject: unknown tag
PS-09|FLOW_LS|210.5|1789990000|10      quarantine: clock skew >300 s
PS-09|FLOW_LS|210.5;DROP|1789999990|11 reject: not numeric
PS-04|LEVEL_M|3.47|1789999999          reject: field count
{'accept': 3, 'reject': 6, 'quarantine': 2}

Before changing anything, write down what you expect each line to print. Then run it. Any line where your expectation was wrong is a place where your mental model of the code was wrong — which is the purpose of the exercise.

Three lines usually surprise readers: nan being rejected rather than stored, PS-7 failing the identifier pattern although it looks harmless, and the 9.80 mg/L chlorine reading ending in quarantine rather than reject. If you predicted any of them wrongly, write down why, then add one bad case of your own to the message list, such as a sequence number lower than 102 or an identifier with a trailing space, and confirm the validator rejects it before you trust the change.

3.7 Common errors and limits

Validation protects the receiver from malformed and implausible data. It does not prove the data are authentic: a perfectly formatted, plausible message can still be forged. Authentication is added in Chapter 15.

Exercises — chapter 3

EX007 — Predict, then run. Objective: test your model of validation order. Inputs: Lab 3. Constraints: write predictions before running. Steps: (1) add three messages: PS-04|LEVEL_M|inf|1789999999|103, PS-04|LEVEL_M|12.0|1789999999|103, PS-04|LEVEL_M|-0.0|1789999999|104; (2) predict each outcome; (3) run and explain any surprise. Expected evidence: predictions, output, explanation. Rubric: 3 = inf rejected as not finite, 12.0 accepted (inclusive limit), -0.0 accepted because −0.0 equals 0.0 and the sequence advanced, with a comment on whether −0.0 should be normalized; 2 = outcomes right, explanation thin; 1 = not run; 0 = none.

EX008 — Quarantine design. Objective: design the quarantine path. Inputs: Chapter 3 ("Reject, quarantine, accept") and the PS-07 chlorine case. Constraints: no code required. Steps: (1) say where quarantined messages are stored and for how long; (2) say who reviews them, how quickly, and what triggers escalation to Water operations; (3) say what the operator display should show while a value is quarantined. Expected evidence: half-page design. Rubric: 3 = separate store with retention, named reviewer and response time, escalation rule tied to the physical process (for example two consecutive implausible chlorine readings → call the control room), display shows "value withheld — under review" rather than the last good value as if current; 2 = one element missing; 1 = "an admin checks it"; 0 = none.

EX009 — Fail closed. Objective: find a fail-open path. Inputs: this fragment: try: v = float(val) except: pass; if v > hi: return "quarantine"; return "accept". Constraints: identify, explain, fix. Steps: (1) show an input for which it returns accept wrongly; (2) explain; (3) rewrite it. Expected evidence: input, explanation, corrected code. Rubric: 3 = shows that a non-numeric value leaves v unset or stale (a NameError on the first message, or the previous message's value reused later), notes the missing lower bound and the bare except, and rewrites to return reject on any parse failure with both bounds and a finiteness check; 2 = finds one issue; 1 = vague; 0 = none.

Chapter synthesis

Small code decides what operators see. Treat every message as untrusted, validate it in layers — structure, identity, type, physical meaning, time, order — and give borderline data a quarantine path so that faults are neither hidden nor stored as truth. Keep decisions in small testable functions, make state explicit, fail closed and test bad cases first. Validation is not authentication; that comes later.

Continue studying — chapter 03 of 30

← Chapter 02 · Course chapter index · Chapter 04 →

Continue to the next authored lesson.

Worked answers

EX007 — Predict, then run. Output for the three added lines: inf → reject: not finite; 12.0 → accept (the range check is inclusive); -0.0 → accept (−0.0 equals 0.0, so it is within range, and sequence 104 is greater than 103). Counts become {'accept': 5, 'reject': 7, 'quarantine': 2}. A good comment notes that −0.0 is harmless but should be normalized to 0.0 before storage so that later comparisons and displays are consistent, and that an exactly-at-limit level of 12.0 m might itself deserve review if 12.0 is the overflow height.

EX008 — Quarantine design. Quarantined messages go to a separate table with the raw message, reason and time, kept 90 days (longer if part of an incident). FSTS monitoring reviews them within one working hour in the day and by the on-call at night when the rule below fires. Escalation: two or more consecutive implausible chlorine or level readings from one station, or any implausible chlorine reading during a dosing change window, triggers a call to the Water control room. The operator display shows "value withheld — under review" with the age of the last good value, never the last good value as if current. Full credit ties escalation to the physical process and owns every step.

EX009 — Fail closed. With the bare except: pass, a non-numeric value such as abc leaves v unset on the first message (a NameError, which may crash the relay — a denial of service) or, in a loop, keeps the previous message's value, which is then checked and possibly accepted as if it belonged to this message. There is also no lower bound, so negative values are accepted, and nan passes because nan > hi is false. Corrected: try: v = float(val) except ValueError: return "reject: not numeric"; if not math.isfinite(v): return "reject: not finite"; if not lo <= v <= hi: return "quarantine: out of physical range"; then later checks; final return "accept" only after all checks.