◇ OpenCyber Academy
Engineering · Chapter 9 · English

Chapter 9 — Programming II

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 9 — Programming II

Lesson map

Lesson 09 published. Map, example, EX025.

flowchart TD
  N0["Learning outcomes"]
  N1["9.1 Pipelines"]
  N2["9.2 Errors as data"]
  N3["9.3 Safe JSON"]
  N4["9.4 Properties"]
  N5["9.5 Versioning parsers"]
  N6["Practice note — Backpressure"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX025"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["9.1 Pipelines"]
  N2["9.2 Errors as data"]
  N3["9.3 Safe JSON"]
  N4["9.4 Properties"]
  N5["9.5 Versioning parsers"]
  N6["Practice note — Backpressure"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX025"]

[Learning outcomes]
    |
    v
[9.1 Pipelines]
    |
    v
[9.2 Errors as data]
    |
    v
[9.3 Safe JSON]
    |
    v
[9.4 Properties]
    |
    v
[9.5 Versioning parsers]
    |
    v
[Practice note — Backpressure]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX025]

Learning outcomes

You will (1) structure a parser as a pipeline of stages; (2) handle errors without losing records silently; (3) parse JSON safely with the standard library; (4) write property-style tests for normalisation.

9.1 Pipelines

Lantern's ingestion is a chain: fetch → decode → validate → normalise → deduplicate → store. Each stage takes records and yields records plus rejects. Generators in Python let large feeds stream without loading everything into memory.

Each stage can be tested on its own with a short list of records, and a failure in one stage does not hide what the others did. Kofi's logs show how many records entered and left every stage, so a sudden drop points to the stage that caused it. When a partner sends a very large archive, streaming means memory use stays flat rather than growing with the file.

9.2 Errors as data

A malformed object is not an exception to swallow; it is a reject record with a reason, the source and the original bytes' hash. Reject counts per source become a data-quality metric that partners can act on.

Kofi's reject records keep the hash rather than the content, so a partner's restricted data is not copied into logs that wider staff can read. If the same reason appears for many records from one source, the partner probably changed something on their side. The report sent to them gives counts and examples, which is usually enough for them to find and fix the problem.

9.3 Safe JSON

json.loads parses data without executing code, unlike evaluating text. Limit input sizes to avoid memory exhaustion and validate types after parsing; JSON numbers, strings and nulls arrive in unexpected places.

Kofi sets a maximum size for each object before parsing, so a huge input is rejected instead of filling memory. After parsing he checks that each field has the expected type: a timestamp field holding a number, or a list where a string was expected, is a reject with a clear reason. Older code that evaluated text as a program to save effort was removed during his first month.

9.4 Properties

Instead of only example tests, state properties: normalising twice equals normalising once (idempotence); output is always lower case; no output contains whitespace at either end. Generate many random inputs and check the properties hold.

Kofi's random inputs include mixed case, leading and trailing spaces, empty strings and characters from other alphabets. A property test found that his domain normaliser left a trailing dot in place, which made two forms of the same name count as different indicators. Fixing it removed a few hundred duplicates from the store.

9.5 Versioning parsers

When a partner changes its export format, a new parser version is deployed alongside the old one until the switch is verified. Each stored object records the parser version that produced it.

Running both versions on the same input for a few days shows whether the new one produces the same records, more rejects or different values. Only when the differences are understood does the old parser stop. Recording the version on each object means that if a bug is found later, the affected records can be identified and reprocessed rather than guessed.

Practice note — Backpressure

When a downstream stage slows, upstream stages must not keep pushing until memory runs out. Bounded queues between stages give backpressure: producers wait when a queue is full. Lantern's collector pauses polling when the validation queue is at capacity and records the pause, so operators see it rather than discovering missing data later.

Worked example

A partner sends 5,000 objects with 37 malformed timestamps. Kofi's pipeline stores 4,963, writes 37 rejects with reason "created not RFC 3339", and emails the partner's data-quality contact a count and two example hashes — not the raw objects, which are TLP:AMBER.

Synthetic case. Counts and the partner are invented.

Errors and limits

Silently dropping bad records hides partner problems; silently fixing them invents data. Do not evaluate untrusted text as code.

Exercises

Synthesis

Stream through stages, treat errors as data, parse safely, test properties and record parser versions.

Continue studying — chapter 09 of 50

← Chapter 08 (draft) · Course chapter index · Chapter 10 (draft) →