Chapter 4 — Scripting and data formats | Technical Application and software supply chain security

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 4 — Scripting and data formats

Lesson map

Lesson 04 published. Map, example, EX013.

flowchart TD
  N0["Learning outcomes"]
  N1["4.1 Hashes"]
  N2["Lab — download verification"]
  N3["4.2 Lab result"]
  N4["4.3 Where hashes come from"]
  N5["4.4 Data formats"]
  N6["4.5 Safe scripts"]
  N7["Reading the script"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX013"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["4.1 Hashes"]
  N2["Lab — download verification"]
  N3["4.2 Lab result"]
  N4["4.3 Where hashes come from"]
  N5["4.4 Data formats"]
  N6["4.5 Safe scripts"]
  N7["Reading the script"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX013"]

[Learning outcomes]
    |
    v
[4.1 Hashes]
    |
    v
[Lab — download verification]
    |
    v
[4.2 Lab result]
    |
    v
[4.3 Where hashes come from]
    |
    v
[4.4 Data formats]
    |
    v
[4.5 Safe scripts]
    |
    v
[Reading the script]
    |
    v
[Worked example] --> [Exercise EX013]

Learning outcomes

You will (1) compute and compare file hashes; (2) read simple JSON and CSV records; (3) write small safe stdlib scripts; (4) verify downloads before installing.

4.1 Hashes

A cryptographic hash such as SHA-256 is a short fingerprint of a file. Any change to the file, even one byte, produces a completely different hash. Vendors publish hashes so users can check a download is exactly the released file. For Ferncroft, the vendor's published hash for resize-plugin 2.4.1 would have differed from the mirror copy's hash. Checking takes less than a minute, while the redirects ran for weeks. The vendor's hash for resize-plugin 2.4.1 is exactly what the lab below compares against each downloaded copy.

Lab — download verification

"""Verify synthetic downloaded files against a published SHA-256 manifest (stdlib only)."""
import hashlib
files = {  # synthetic file contents
    "resize-plugin-2.4.1.zip": b"synthetic resize plugin 2.4.1 official",
    "gallery-plugin-1.8.0.zip": b"synthetic gallery plugin 1.8.0 official",
    "resize-plugin-2.4.1-mirror.zip": b"synthetic resize plugin 2.4.1 modified",
}
manifest = {  # published by the synthetic vendor
    "resize-plugin-2.4.1": hashlib.sha256(files["resize-plugin-2.4.1.zip"]).hexdigest(),
    "gallery-plugin-1.8.0": hashlib.sha256(files["gallery-plugin-1.8.0.zip"]).hexdigest(),
}
for name, data in files.items():
    key = name.replace("-mirror", "").removesuffix(".zip")
    got = hashlib.sha256(data).hexdigest()
    status = "MATCH" if manifest.get(key) == got else "MISMATCH - do not install"
    print(f"{name:32} {got[:12]}  {status}")

Run with python3 lab_hash.py. Verified output:

resize-plugin-2.4.1.zip          e85dc388d054  MATCH
gallery-plugin-1.8.0.zip         80324825a8be  MATCH
resize-plugin-2.4.1-mirror.zip   5af89d46d94f  MISMATCH - do not install

4.2 Lab result

The official resize-plugin-2.4.1.zip and gallery-plugin-1.8.0.zip match the vendor manifest. resize-plugin-2.4.1-mirror.zip has a different hash (beginning 5af89d46d94f) and is reported as MISMATCH - do not install. This is exactly the check that would have stopped the Ferncroft installation. The lab compares full hashes even though it prints only the first twelve characters. A mismatch is never a judgment call: the file is not the released one, whatever the mirror claims about it. Grace's new ticket field later records this result for every installation that support performs.

4.3 Where hashes come from

A hash is only useful if it comes from a trusted source, such as the vendor's own site over HTTPS or a signed release. A hash published next to a file on the same mirror proves nothing, because whoever changed the file could change the hash too. For the resize plugin, the trusted source is the vendor's release page reached over HTTPS, not the mirror where the previous technician downloaded the file. Grace's new ticket field records where the hash came from, so a reviewer can check that the source was the vendor's own page.

4.4 Data formats

Manifests and advisories are often JSON or CSV. Python's json and csv modules read them safely without extra packages. Reading these formats with the standard library avoids installing extra packages on support machines. The same json and csv modules later read the advisory feeds and the nightly inventory CSV, so Tomás learns one set of tools that he uses throughout the course and in the nightly job. Reading a file never runs code from it, which keeps these scripts safe on support machines.

4.5 Safe scripts

Support scripts only read data and print results; any change to a customer site goes through a ticket. A read-only script cannot damage a customer site even if it contains a bug. The download-verification lab follows this rule: it computes hashes and prints MATCH or MISMATCH, but installing or rejecting the file is a decision recorded in the ticket. The nightly job built in Chapter 14 later keeps exactly the same read-only rule. Separating reading from changing also lets new technicians run the scripts without extra permissions.

Reading the script

The lab computes each file's SHA-256 with hashlib and compares it to the manifest entry for the release name. It prints only the first twelve characters for readability, but compares the full value.

Worked example

Grace makes hash verification a required step: the support ticket for a plugin installation now has a field for the vendor hash source and the computed hash. In the first month, the step catches one more mismatched download from a slow mirror.

Synthetic case. Files and hashes are synthetic.

Errors and limits

A matching hash proves the file is the released one, not that the release is free of flaws.

Exercises

EX013 — Lab result. Which file failed and what does the lab advise? Rubric: 3 = mirror zip; do not install; 2 = partial; 1 = vague; 0 = none.
EX014 — Hash source. Why must the hash come from a trusted source? Rubric: 3 = attacker could change both; 2 = partial; 1 = vague; 0 = none.
EX015 — What a match proves. What does a matching hash prove? Rubric: 3 = file is the released one, not flawless; 2 = partial; 1 = vague; 0 = none.
EX016 — New step. What did Grace add to the ticket? Rubric: 3 = hash source and computed hash fields; 2 = partial; 1 = vague; 0 = none.

Synthesis

Verify downloads against trusted hashes before installing.

Continue studying — chapter 04 of 18

← Chapter 03 (draft) · Course chapter index · Chapter 05 (draft) →