OpenCyber Academy
Technologist · Chapter 19 · English
Course code: OCA-083Chapter 19: OCA-083-C019

Chapter 19 — Application and software supply chain security: monitoring

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 19 published. Map, example, EX055.

flowchart TD
  N0["Learning outcomes"]
  N1["19.1 Drift"]
  N2["Lab — release drift"]
  N3["19.2 Lab result"]
  N4["19.3 Investigating"]
  N5["19.4 Alerts"]
  N6["19.5 Closing the loop"]
  N7["Tuning alerts"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX055"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["19.1 Drift"]
  N2["Lab — release drift"]
  N3["19.2 Lab result"]
  N4["19.3 Investigating"]
  N5["19.4 Alerts"]
  N6["19.5 Closing the loop"]
  N7["Tuning alerts"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX055"]

[Learning outcomes]
    |
    v
[19.1 Drift]
    |
    v
[Lab — release drift]
    |
    v
[19.2 Lab result]
    |
    v
[19.3 Investigating]
    |
    v
[19.4 Alerts]
    |
    v
[19.5 Closing the loop]
    |
    v
[Tuning alerts]
    |
    v
[Worked example] --> [Exercise EX055]

Learning outcomes

You will (1) compare running versions with approved releases; (2) detect drift and unapproved builds; (3) alert owners; (4) investigate differences.

19.1 Drift

Drift is when what runs differs from what was approved: a client left behind on an old release, or something running that the pipeline never approved. Both kinds appear in the lab: harbour-cafe left behind on an older release, and museum-tickets running a build the pipeline never approved. Drift matters because every earlier control assumes that what runs is what was built and approved. A client left behind keeps old library versions after the audit says they were fixed, while an unapproved build bypasses review, pinning and digest records entirely.

Lab — release drift

"""Compare deployed versions with the latest approved release per client (synthetic)."""
approved = {"greenleaf-booking": "5.3.0", "harbour-cafe": "2.1.4", "kiteschool-site": "1.9.0", "museum-tickets": "4.0.2"}
deployed = {"greenleaf-booking": "5.3.0", "harbour-cafe": "2.0.9", "kiteschool-site": "1.9.0", "museum-tickets": "4.0.3-hotfix"}
for c in sorted(approved):
    d, a = deployed[c], approved[c]
    state = "ok" if d == a else ("UNAPPROVED BUILD" if "-" in d else "BEHIND")
    print(f"{c:18} deployed {d:13} approved {a:6} {state}")

Run with python3 lab_drift.py. Verified output:

greenleaf-booking  deployed 5.3.0         approved 5.3.0  ok
harbour-cafe       deployed 2.0.9         approved 2.1.4  BEHIND
kiteschool-site    deployed 1.9.0         approved 1.9.0  ok
museum-tickets     deployed 4.0.3-hotfix  approved 4.0.2  UNAPPROVED BUILD

19.2 Lab result

harbour-cafe runs 2.0.9 while 2.1.4 is approved, so it is BEHIND. museum-tickets runs 4.0.3-hotfix, which is not in the approved list, so it is an UNAPPROVED BUILD. Other clients match. The lab treats any version containing a hyphen as an unapproved build, which works for these labels but is a simplification. A real check compares digests with the approved list rather than reading version strings. Reading the output, Diego starts with the unapproved build, because nobody yet knows what it contains. A real check would therefore not depend on how someone chose to name a build.

19.3 Investigating

BEHIND usually means a deployment failed or was skipped; UNAPPROVED BUILD needs urgent questions about where the build came from. For harbour-cafe, the cause was a failed database migration that stopped the deployment. For museum-tickets, the cause was a manual weekend build deployed during an outage before the approved-image rule. For a BEHIND client, the owner checks the deployment log and reruns the failed step. For an UNAPPROVED BUILD, Aiko asks who built it, from which code and why, before deciding whether to replace it.

19.4 Alerts

A daily check sends drift to the project owner and, for unapproved builds, to Aiko too. Alerts name the client, both versions and the status, so the owner can start without opening the full report. Holds agreed with a client and recorded with an end date stay quiet until that date passes. The daily timing means drift is found within a day of appearing rather than at the next manual check. Sending unapproved builds to Aiko as well reflects that their origin is unknown.

19.5 Closing the loop

Each finding is recorded with its cause and fix. Recording causes over time shows patterns, such as failed migrations, that point to fixes beyond the single case. Recording the cause matters because a fix without a cause can return. When harbour-cafe's deployment stalls on a failed migration, the record names the migration and the fix, so a similar stall elsewhere is recognised quickly. The museum-tickets record notes the manual weekend deployment and its replacement by a pipeline build of the same change. Over time the records show which causes recur and which earlier controls have stopped them.

Tuning alerts

At first, the daily check alerts on every client behind the latest release, including those deliberately held back while a client tests a change. Diego adds an approved hold list with an end date, so expected differences stay quiet while unexpected ones still alert. Holds that pass their end date alert again. This keeps owners trusting the alerts, because almost every alert now needs action. Over the following month, the check raises nine alerts; eight lead to fixes and one to a hold being extended with the client's agreement.

Reading the lab

The lab compares a list of running versions with the approved list and prints a status per client, quoting both versions.

Worked example

The museum-tickets hotfix turns out to have been built and deployed manually during a weekend outage before the approved-image rule. It is replaced by a pipeline build of the same change, and the harbour-cafe deployment, stuck on a failed migration, is fixed.

Synthetic case. Versions and causes are synthetic.

Errors and limits

Monitoring shows differences; people must find the causes.

Exercises

Synthesis

Watch for drift and fix its causes.

Continue studying — chapter 19 of 30

← Chapter 18 (draft) · Course chapter index · Chapter 20 (draft) →

Worked answers

EX055. harbour-cafe BEHIND (2.0.9 vs 2.1.4) and museum-tickets UNAPPROVED BUILD (4.0.3-hotfix).

EX056. Nobody can say how it was built or what it contains.

EX057. A manual weekend deployment made before the approved-image rule.