Chapter 10 — Application and software supply chain security: quality and troubleshooting
Lesson map
Lesson 10 published. Map, example, EX046.
flowchart TD
N0["Learning outcomes"]
N1["10.1 Measuring"]
N2["Lab — remediation targets"]
N3["10.2 Lab result"]
N4["10.3 Finding causes"]
N5["10.4 Improvements"]
N6["10.5 Troubleshooting tools"]
N7["Reading the lab"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX046"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["10.1 Measuring"]
N2["Lab — remediation targets"]
N3["10.2 Lab result"]
N4["10.3 Finding causes"]
N5["10.4 Improvements"]
N6["10.5 Troubleshooting tools"]
N7["Reading the lab"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX046"]
[Learning outcomes]
|
v
[10.1 Measuring]
|
v
[Lab — remediation targets]
|
v
[10.2 Lab result]
|
v
[10.3 Finding causes]
|
v
[10.4 Improvements]
|
v
[10.5 Troubleshooting tools]
|
v
[Reading the lab]
|
v
[Worked example] --> [Exercise EX046]
Learning outcomes
You will (1) measure whether the workflow meets its targets; (2) find where it fails; (3) troubleshoot slow fixes; (4) improve the process.
10.1 Measuring
Priya measures how many findings were fixed within target, by severity, from the ticket records. Ticket records show when each finding was created, ranked and closed after the scanner re-run, so time to fix can be calculated without extra reporting. Measuring by severity shows whether the most important findings are fixed fastest. Priya counts findings as on time if they closed within their target, and records the definition so later figures are comparable. Reopened tickets count from their first creation date, so a fix that failed the scanner re-run cannot reset the clock.
Lab — remediation targets
"""Measure remediation within target days per severity on synthetic records (stdlib only)."""
targets = {"critical": 7, "high": 30, "medium": 90}
records = [("critical", 3), ("critical", 9), ("high", 12), ("high", 25), ("high", 41), ("high", 18),
("medium", 60), ("medium", 120), ("medium", 45), ("medium", 80)]
for sev, t in targets.items():
days = [d for s, d in records if s == sev]
ok = sum(d <= t for d in days)
print(f"{sev:8} target {t:>3}d: {ok}/{len(days)} on time ({100*ok/len(days):.0f}%), longest {max(days)}d")
Run with python3 lab_sla.py. Verified output:
critical target 7d: 1/2 on time (50%), longest 9d
high target 30d: 3/4 on time (75%), longest 41d
medium target 90d: 3/4 on time (75%), longest 120d
10.2 Lab result
Critical: 1 of 2 on time (50%), longest 9 days against a 7-day target. High: 3 of 4 on time (75%), longest 41 days against 30. Medium: 3 of 4 on time (75%), longest 120 days against 90. The critical miss matters most despite the small numbers. The numbers are small, so percentages change sharply with one finding. Priya reports the counts beside each percentage, so readers do not overread them. The critical miss matters because criticals carry the most risk, even though only two occurred. She presents the counts to Marcus and the product manager with the causes of each miss.
10.3 Finding causes
Priya reviews each missed target. The late critical waited two days for a platform change; the late high was the blocked framework upgrade; the late medium was forgotten after its owner moved teams. Each miss had a different cause, which points to a different fix. The late critical shows a dependency on another team's speed; the late high shows a blocked item without an early risk decision; the late medium shows that tickets can be lost when people move. Priya records each cause with its evidence from the ticket history, without blaming the people involved.
10.4 Improvements
Platform requests for criticals get a same-day contact; ownership transfers when people move; blocked items get risk acceptances earlier. Each improvement answers one cause. A same-day platform contact means criticals are not left waiting for another team's queue. Transferring ownership when people move keeps tickets alive. Earlier risk acceptances make blocked items visible and controlled rather than quietly late. In the next quarter, all criticals were fixed on time and highs reached 90% on time, with small sample sizes stated. Each improvement is recorded with an owner and a review date, so Priya can check next quarter whether it was actually used.
10.5 Troubleshooting tools
When scanners produce odd results, Priya checks versions of the scanner and its advisory data first, then the project's lock file. Odd results often come from outdated advisory data or a scanner update rather than a change in TrackPort. Checking versions first avoids chasing problems that do not exist. The lock file then shows exactly which package versions the scanner saw. Priya records what she checked and what she found, so the same problem can be diagnosed faster next time.
Reading the lab
The lab groups records by severity, counts those within target, and prints the longest time. Priya adds the longest time because averages hide the cases that matter most, such as a critical left open for days beyond its target. She also tries changing the critical target to 10 days to show the product manager how much a target choice changes the result, which helps the discussion about what targets should be. Both runs are kept with their settings.
Quarterly review
Every quarter the target results and causes are reviewed with the lead and product manager.
Worked example
The next quarter, with the improvements in place, all criticals are fixed on time and highs reach 90% on time. Priya reports the numbers with their small sample sizes clearly stated.
Synthetic case. Records are synthetic.
Errors and limits
Small numbers swing widely; look at causes as well as percentages.
Exercises
- EX046 — Lab critical. What was the critical result? Rubric: 3 = 1/2, longest 9 days; 2 = partial; 1 = vague; 0 = none.
- EX047 — Lab high. What was the high result? Rubric: 3 = 3/4, longest 41; 2 = partial; 1 = vague; 0 = none.
- EX048 — Causes. Name the causes of misses. Rubric: 3 = platform wait; blocked upgrade; owner moved; 2 = two; 1 = one; 0 = none.
- EX049 — Fixes. Name two improvements. Rubric: 3 = same-day contact; ownership transfer; early acceptance; 2 = one; 1 = vague; 0 = none.
- EX050 — Small numbers. Why state sample sizes? Rubric: 3 = percentages swing with few items; 2 = partial; 1 = vague; 0 = none.
Synthesis
Measure targets, find causes and improve the workflow.
Continue studying — chapter 10 of 12
← Chapter 09 (draft) · Course chapter index · Chapter 11 (draft) →