Chapter 4 — Application and software supply chain security: competing explanations
Lesson map
Lesson 04 published. Map, example, EX021.
flowchart TD
N0["Learning outcomes"]
N1["4.1 The worry"]
N2["4.2 Pentest effort"]
N3["4.3 Framework change"]
N4["4.4 People"]
N5["4.5 Tester focus"]
N6["4.6 Weighing"]
N7["Unchecked explanations"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX021"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["4.1 The worry"]
N2["4.2 Pentest effort"]
N3["4.3 Framework change"]
N4["4.4 People"]
N5["4.5 Tester focus"]
N6["4.6 Weighing"]
N7["Unchecked explanations"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX021"]
[Learning outcomes]
|
v
[4.1 The worry]
|
v
[4.2 Pentest effort]
|
v
[4.3 Framework change]
|
v
[4.4 People]
|
v
[4.5 Tester focus]
|
v
[4.6 Weighing]
|
v
[Unchecked explanations]
|
v
[Worked example] --> [Exercise EX021]
Learning outcomes
You will (1) identify alternative explanations for a security improvement; (2) design checks for each; (3) weigh evidence fairly; (4) state what remains uncertain.
4.1 The worry
If access-control findings fall after tests are added, other things might explain it: fewer pentest days, a new framework, a senior developer joining, or testers focusing elsewhere.
Each alternative predicts a different pattern. Fewer pentest days would lower all findings, not just access control. A new framework would lower findings mainly in the routes it changed. A senior developer would show up as a change in review practice. Tester focus elsewhere would appear in the pentest scopes. Listing them before the results lets each be checked against data.
4.2 Pentest effort
Sofia records tester-days per release. They were stable (four days each), so reduced effort does not explain a fall.
Stable effort rules out the simplest alternative. If tester-days had fallen from four to two after the tests were introduced, fewer findings would be expected whatever the tests did, and the comparison would need to be adjusted for effort. Recording tester-days per release is cheap and closes that question for all twelve releases. Effort also stays comparable across the two periods because the same pentest supplier ran every test; Sofia records the supplier with the tester-days so this is checkable.
4.3 Framework change
In release 8 Tallyline upgraded its web framework, which added a default deny for unmapped routes. This could reduce flaws by itself. Sofia separates findings in routes the upgrade affected from others.
Separating the routes gives a cleaner test. In routes the upgrade affected, the default deny and the tests could both explain a fall. In routes the upgrade did not affect, only the tests (and the other alternatives) could. Sofia found that findings still fell sharply in the unaffected routes, which supports a real effect of the tests, though smaller than the raw comparison suggests.
4.4 People
A senior developer joined at release 7. Sofia notes the date and asks reviewers whether review practice changed; it did, slightly.
Release 7 is one release before the tests and the framework upgrade, so a change in review practice would overlap with both. Sofia cannot separate its effect statistically with twelve releases. She records it as a possible contributor and lets the reviewers' account of a slight change stand as the evidence. Asking reviewers rather than inferring from the data is deliberate: twelve releases cannot separate a person's effect statistically, but reviewers can say what they did differently. Their answer, a slight change, is reported as it was given.
4.5 Tester focus
Pentest scopes are checked: authorisation was in scope for all twelve releases.
That check matters because a fall in access-control findings could simply mean the testers stopped looking for them. Since authorisation stayed in scope throughout, a release with no access-control findings means testers looked and found none, not that they did not look. The same four tester-days per release support this. Sofia checks the scope documents for all twelve releases rather than a sample, because a single release where authorisation was dropped from scope would produce a misleading zero. The scope check takes an afternoon and closes the question fully.
4.6 Weighing
Some of the improvement may come from the framework default and the new developer; the tests are one cause among several. Sofia plans to report this honestly rather than credit everything to tests.
Weighing the alternatives together leads to a ranked account rather than a single cause: the tests have the strongest support, because findings fell even in routes the framework change did not affect; the framework default explains part of the fall in the routes it did affect; and the senior developer may have contributed through slightly changed review practice.
Unchecked explanations
Sofia could not measure developer awareness directly, so she lists it as an unchecked explanation. A short survey before and after might address it in future studies, and she notes it as a recommendation.
Diagram of causes
Sofia draws a simple diagram linking tests, the framework change, the new developer and pentest effort to the outcome. The diagram shows which paths she can measure and which she cannot, and it helps readers see why she separated framework-affected routes in the analysis.
Worked example
Separating findings, Sofia finds that in routes unaffected by the framework change, findings still fell sharply. This supports a real effect of the tests, though smaller than the raw comparison suggests.
Synthetic case. Releases, staff and findings are synthetic.
Errors and limits
Unmeasured explanations may remain; say which you could not check.
Exercises
- EX021 — Four alternatives. List four alternative explanations. Rubric: 3 = pentest effort, framework, new developer, tester focus; 2 = three; 1 = one; 0 = none.
- EX022 — Effort check. How was pentest effort checked? Rubric: 3 = tester-days stable at four; 2 = partial; 1 = vague; 0 = none.
- EX023 — Framework effect. What did the framework upgrade add? Rubric: 3 = default deny for unmapped routes; 2 = partial; 1 = vague; 0 = none.
- EX024 — Separation result. What did separating routes show? Rubric: 3 = unaffected routes still fell; real but smaller effect; 2 = partial; 1 = vague; 0 = none.
- EX025 — Scope check. Was authorisation always in scope? Rubric: 3 = yes, all twelve releases; 2 = one; 1 = vague; 0 = none.
- EX026 — Honest reporting. How will Sofia report causes? Rubric: 3 = tests one cause among several; 2 = partial; 1 = vague; 0 = none.
Synthesis
Name rival explanations, check each, and report the share of credit honestly.
Continue studying — chapter 04 of 08
← Chapter 03 (draft) · Course chapter index · Chapter 05 (draft) →