Chapter 20 — Application and software supply chain security: troubleshooting
Lesson map
Lesson 20 published. Map, example, EX058.
flowchart TD
N0["Learning outcomes"]
N1["20.1 Common failures"]
N2["20.2 Diagnosing"]
N3["20.3 Not disabling checks"]
N4["20.4 Breaking changes"]
N5["20.5 Recording lessons"]
N6["20.6 Knowing when to ask"]
N7["The troubleshooting page"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX058"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["20.1 Common failures"]
N2["20.2 Diagnosing"]
N3["20.3 Not disabling checks"]
N4["20.4 Breaking changes"]
N5["20.5 Recording lessons"]
N6["20.6 Knowing when to ask"]
N7["The troubleshooting page"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX058"]
[Learning outcomes]
|
v
[20.1 Common failures]
|
v
[20.2 Diagnosing]
|
v
[20.3 Not disabling checks]
|
v
[20.4 Breaking changes]
|
v
[20.5 Recording lessons]
|
v
[20.6 Knowing when to ask]
|
v
[The troubleshooting page]
|
v
[Worked example] --> [Exercise EX058]
Learning outcomes
You will (1) diagnose failing dependency updates and pipeline checks; (2) avoid disabling checks; (3) fix root causes; (4) record lessons.
20.1 Common failures
Update pull requests fail tests because of breaking changes; pinned actions break when their own dependencies change; content checks reject unusual but valid files; audit matches turn out false. Each failure has a tempting shortcut: skipping the update, unpinning the action, widening the upload rules or ignoring the audit. The troubleshooting work in this chapter is about finding the cause instead. Diego keeps a list of these failure types so new ones can be compared with what the team has already solved.
20.2 Diagnosing
Diego reads the failure, reproduces it locally with synthetic data, and changes one thing at a time. Reproducing with synthetic data keeps real client data out of debugging and lets the failure be run again on demand. Changing one thing at a time means that when the failure stops, Diego knows which change fixed it. Writing down each change and its effect helps if the problem returns later. If the failure cannot be reproduced locally, that difference is itself a clue, often pointing to configuration on the build machine.
20.3 Not disabling checks
A failing check is never switched off to make a deadline. If it must be bypassed, Aiko approves a time-limited exception with a ticket. The ticket records which check, which project, why it is bypassed and the expiry date. When the date passes, the exception ends automatically and the check runs again. Exceptions are listed in the monthly report, so expired or repeated ones are noticed. The rule exists because a disabled check rarely gets switched back on once the deadline has passed and attention moves elsewhere.
20.4 Breaking changes
For a major library update, the developer reads the change notes and updates code in a separate pull request. Keeping code changes separate from the version bump lets reviewers see which edits the new version required. If the update later causes trouble, the previous approved digest can be redeployed while the code change is investigated. The bot's minor updates stay small, while major ones get their own planned pull request. This keeps the weekly bot's small updates moving while bigger changes get the attention they need.
20.5 Recording lessons
Each solved problem gets a short note in the team's troubleshooting page. The date-library time-zone note is the clearest payoff: a month later it saved another developer an afternoon on the same failure. Each note uses the same four fields: symptom, cause, fix and date. Diego writes the note on the day the problem is solved, while the details are fresh, and links it to the pull request that fixed it. The quarterly review removes notes about replaced libraries, so search results stay relevant. Searching the page is the first step when a check fails, before anyone starts debugging from scratch.
20.6 Knowing when to ask
If a problem takes more than an hour, Diego asks for help. Asking early also spreads knowledge, because the person helping learns about the failure too. Pair debugging often finds the cause quickly, usually because one stated assumption turns out to be wrong. The one-hour rule protects deadlines and stops a student from spending a day stuck on a failure that a colleague has already seen. Diego brings what he has tried, the error and his current guess, so the person helping can start quickly. The answer usually becomes a troubleshooting note for the next developer.
The troubleshooting page
Each note has the symptom, the cause, the fix and the date. Notes are searchable by library or check name. Diego reviews the page every quarter and removes notes that no longer apply, for example after a library is replaced. He notices that a third of the notes concern one legacy project, which strengthens the case to rebuild it.
Pair debugging
For harder failures, Diego works with another developer: one describes the failure aloud while the other questions each assumption. Several stubborn problems were solved this way within minutes, usually because a stated assumption turned out wrong.
Worked example
A weekly update fails because a date library changed its time-zone default, breaking booking times in tests. Diego sets the time zone explicitly, the tests pass, and the troubleshooting page saves another developer an afternoon a month later.
Synthetic case. Failures are synthetic.
Errors and limits
Exceptions must expire; permanent exceptions become hidden risk.
Exercises
- EX058 — No disabling. What happens if a check must be bypassed? Rubric: 3 = time-limited approved exception; 2 = partial; 1 = vague; 0 = none.
- EX059 — Diagnosis steps. How does Diego diagnose? Rubric: 3 = read, reproduce, one change at a time; 2 = two; 1 = one; 0 = none.
- EX060 — Date library fix. What fixed the date library failure? Rubric: 3 = explicit time zone; 2 = partial; 1 = vague; 0 = none.
Synthesis
Fix root causes and keep checks on.
Continue studying — chapter 20 of 30
← Chapter 19 (draft) · Course chapter index · Chapter 21 (draft) →