Chapter 4 — Data models
Lesson map
Lesson 04 published. Map, example, EX010.
flowchart TD
N0["Learning outcomes"]
N1["4.1 Manifests as data"]
N2["4.2 Version ranges"]
N3["Lab — who is affected?"]
N4["4.3 Lab result"]
N5["4.4 Keeping current"]
N6["4.5 Data quality"]
N7["The inventory schema"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX010"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["4.1 Manifests as data"]
N2["4.2 Version ranges"]
N3["Lab — who is affected?"]
N4["4.3 Lab result"]
N5["4.4 Keeping current"]
N6["4.5 Data quality"]
N7["The inventory schema"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX010"]
[Learning outcomes]
|
v
[4.1 Manifests as data]
|
v
[4.2 Version ranges]
|
v
[Lab — who is affected?]
|
v
[4.3 Lab result]
|
v
[4.4 Keeping current]
|
v
[4.5 Data quality]
|
v
[The inventory schema]
|
v
[Worked example] --> [Exercise EX010]
Learning outcomes
You will (1) model project dependencies as structured data; (2) query manifests across projects; (3) answer "who uses this version?"; (4) keep the data current.
4.1 Manifests as data
Each project has a manifest listing its libraries and versions. Treated as data, manifests from all projects can be combined into one inventory. In Brightwave's case each manifest is already a small structured file listing names and versions, so the work is mainly collecting them in one place. The lab does exactly that with four synthetic projects and asks one question of the combined data. Scaling the same idea to all 23 client projects is what the nightly inventory does. Each row in the combined data names the project, the library and the installed version.
4.2 Version ranges
Advisories state affected ranges, such as "at least 2.0.0 and below 3.2.0". Versions must be compared as numbers, part by part, not as text. Comparing as text gives wrong answers: "3.10.0" sorts before "3.2.0" because the character 1 comes before 2. The lab splits each version at the dots and compares the parts as integers, so 2.9.8 correctly falls inside the range and 3.4.2 falls outside it. The same comparison runs in the audit lab later, so getting it right once protects every later check. A version string with letters, such as a hotfix label, would break this simple parser, which is why the drift lab treats such labels separately.
Lab — who is affected?
"""Find which client projects use an affected library version (synthetic manifests)."""
import json
manifests = json.loads('''{"greenleaf-booking": {"upload-kit": "3.1.0", "forms": "2.2.0"},
"harbour-cafe": {"forms": "2.4.1"}, "kiteschool-site": {"upload-kit": "3.4.2"},
"museum-tickets": {"upload-kit": "2.9.8", "forms": "2.4.1"}}''')
def v(s): return tuple(map(int, s.split(".")))
affected = (v("2.0.0"), v("3.2.0")) # >=2.0.0, <3.2.0
for proj, deps in sorted(manifests.items()):
ver = deps.get("upload-kit")
if ver:
hit = affected[0] <= v(ver) < affected[1]
print(f"{proj:18} upload-kit {ver:6} {'AFFECTED' if hit else 'ok'}")
Run with python3 lab_inventory.py. Verified output:
greenleaf-booking upload-kit 3.1.0 AFFECTED
kiteschool-site upload-kit 3.4.2 ok
museum-tickets upload-kit 2.9.8 AFFECTED
4.3 Lab result
Three projects use upload-kit. greenleaf-booking (3.1.0) and museum-tickets (2.9.8) fall in the affected range; kiteschool-site (3.4.2) is above it and safe. harbour-cafe does not use upload-kit. In January this took three days by hand; the script answers in under a second. The lab also shows its own limits, which Diego notes beside the result. It checks only upload-kit and only the four synthetic projects written into it, so a project missing from the data would be missed silently. The answer is only as good as the manifests: an outdated copy of a project's file would give a confident but wrong result. That is why the full inventory reads lockfiles fresh every night.
4.4 Keeping current
The inventory is rebuilt nightly from every repository's lockfile, so it reflects exactly what each project installs. Rebuilding from lockfiles rather than manifests matters because a manifest may allow a range of versions while the lockfile records the one actually installed. The nightly run also catches change: when a project updates upload-kit, the next morning's inventory shows the new version without anyone editing a spreadsheet. A run that fails is reported, so a stale inventory is never mistaken for a current one.
4.5 Data quality
Projects without lockfiles show as "unknown" and are fixed first, since their real versions cannot be known. Diego's full run finds two such projects among the 23, and Aiko adds lockfiles to both that week. Until then their rows are treated as possibly affected by every advisory for their declared libraries, which is the safe assumption. An inventory with unexplained gaps would repeat January's problem in a quieter form, because nobody would know which answers to trust. Each unknown row also names the project owner who must add the missing lockfile.
The inventory schema
Each inventory row has: project, client, library, installed version, direct or transitive, and date checked. Transitive dependencies matter because many flaws arrive through libraries that other libraries bring in. Diego adds a column for the owner of each project, so alerts reach the right developer quickly. He also adds a query that lists every library used by more than five clients, because a flaw in one of those would be the most disruptive.
Answering quickly
With the inventory, Diego can answer three questions for any advisory: which clients, which versions, and who owns each project. Aiko times it on a test advisory: four minutes from advisory to a list of owners notified.
Worked example
Diego runs the full inventory: 23 client projects, 412 distinct library versions. Two projects lack lockfiles; Aiko adds them that week.
Synthetic case. Projects and versions are synthetic.
Errors and limits
Manifests list declared dependencies; code copied directly into projects will not appear.
Exercises
- EX010 — Lab result. Which projects were affected? Rubric: 3 = greenleaf-booking, museum-tickets; 2 = one; 1 = vague; 0 = none.
- EX011 — Text comparison. Why not compare versions as text? Rubric: 3 = wrong ordering e.g. 3.10 vs 3.2; 2 = partial; 1 = vague; 0 = none.
- EX012 — Missing lockfiles. Why fix projects without lockfiles first? Rubric: 3 = real versions unknown; 2 = partial; 1 = vague; 0 = none.
Synthesis
Treat manifests as data so "who is affected?" takes seconds.
Continue studying — chapter 04 of 30
← Chapter 03 (draft) · Course chapter index · Chapter 05 (draft) →