Chapter 6 — Data collection and analysis
Lesson map
Lesson 06 published. Map, example, EX033.
flowchart TD
N0["Learning outcomes"]
N1["6.1 Collection"]
N2["Lab — findings per release"]
N3["6.2 Lab result"]
N4["Lab — rate per KLOC"]
N5["6.3 Rate result"]
N6["6.4 Interpreting"]
N7["6.5 Team observations"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX033"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["6.1 Collection"]
N2["Lab — findings per release"]
N3["6.2 Lab result"]
N4["Lab — rate per KLOC"]
N5["6.3 Rate result"]
N6["6.4 Interpreting"]
N7["6.5 Team observations"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX033"]
[Learning outcomes]
|
v
[6.1 Collection]
|
v
[Lab — findings per release]
|
v
[6.2 Lab result]
|
v
[Lab — rate per KLOC]
|
v
[6.3 Rate result]
|
v
[6.4 Interpreting]
|
v
[6.5 Team observations]
|
v
[Worked example] --> [Exercise EX033]
Learning outcomes
You will (1) collect release-level findings and change size; (2) compute rates and rate ratios; (3) interpret small-sample results carefully; (4) combine numbers with team observations.
6.1 Collection
For each of twelve releases Sofia records access-control findings, lines changed, pentest days and notable events. Two classifiers agree on 46 of 48 findings' categories; the two disagreements are resolved by discussion.
Notable events, such as the framework upgrade in release 8 and the senior developer joining at release 7, are recorded beside the counts so that the time series can be read against them. The classifiers' high agreement means the access-control counts do not depend on who classified the findings. Recording these items for every release, not just the ones where something happened, is what makes the before and after periods comparable.
Lab — findings per release
"""Access-control findings per release before and after authorisation tests were added (synthetic)."""
before = {"r1": 3, "r2": 2, "r3": 4, "r4": 3, "r5": 2, "r6": 3}
after = {"r7": 1, "r8": 0, "r9": 1, "r10": 0, "r11": 1, "r12": 0}
for name, d in (("before", before), ("after", after)):
print(f"{name:6} releases={len(d)} findings={sum(d.values())} mean per release={sum(d.values()) / len(d):.2f}")
Run with python3 lab_regress.py. Verified output:
before releases=6 findings=17 mean per release=2.83
after releases=6 findings=3 mean per release=0.50
6.2 Lab result
Before tests: 17 findings over six releases, 2.83 per release. After: 3 findings, 0.50 per release.
The raw counts are the starting point, not the result. Seventeen findings before and three after is a large fall, but it does not yet account for change size, the framework upgrade or the other alternatives, which is why the next steps normalise by KLOC and separate the routes affected by the upgrade. Expressed per release, that is close to three findings before against one every two releases after. The framework-separated comparison and the KLOC rate then test how much of that fall the tests can claim.
Lab — rate per KLOC
"""Findings per 1,000 changed lines, and the rate ratio after/before (synthetic)."""
before_find, before_kloc = 17, 41.0
after_find, after_kloc = 3, 52.0
rb, ra = before_find / before_kloc, after_find / after_kloc
print(f"before {rb:.3f} per KLOC, after {ra:.3f} per KLOC, rate ratio {ra / rb:.2f}")
Run with python3 lab_rate.py. Verified output:
before 0.415 per KLOC, after 0.058 per KLOC, rate ratio 0.14
6.3 Rate result
Normalised by change size, the rate falls from 0.415 to 0.058 findings per KLOC, a rate ratio of 0.14. The after period actually changed more code (52 versus 41 KLOC), so the fall is not due to quieter releases.
A rate ratio of 0.14 means the after-period rate is about one seventh of the before-period rate. Because the after period changed more code, the normalised fall is if anything larger than the raw fall, which removes "quieter releases" as an explanation. Sofia states it next to the framework-separated result, so the ratio is never read as the tests' effect alone.
6.4 Interpreting
Twelve releases and twenty findings are a small sample; Sofia reports the ratio with its uncertainty and avoids saying "86% reduction caused by tests".
The uncertainty matters because twenty findings split over twelve releases leave many releases with zero or one finding. A single release with three more findings after the tests would change the ratio noticeably. Sofia's wording, "was followed by" and "is consistent with", keeps the claim within what such small counts can support. Sofia's table gives the ratio with an interval rather than a single figure.
6.5 Team observations
In interviews, developers say failing authorisation tests often caught problems before review. One says the matrix forced product managers to decide roles explicitly, which reduced arguments.
The interviews do not measure an effect; they explain a mechanism. Catching problems before review suggests the tests work at the point developers write code, and the product managers' explicit role decisions suggest the matrix also changes how requirements are written. Sofia reports both as developers' accounts, alongside the counts. Sofia reports them as developers' accounts of how the tests worked, not as evidence that findings fell; that evidence comes from the counts.
Uncertainty
With so few findings, Sofia also reports a simple range: if the after period had three more findings by chance, the ratio would still be below 0.3, showing a clear fall.
Release notes
Release notes help match findings to features: the refund flaw in release 5 links to a new refund screen added that release. Such links let Sofia check that findings are attributed to the right release and that no findings belong to old code that was simply tested for the first time, which would inflate the before period.
Worked example
Combining the framework-separated analysis (chapter 4) and rates, Sofia's finding is: access-control findings fell substantially after the tests, part of the fall is explained by the framework default, and developers credit the tests with catching problems earlier.
Synthetic case. All counts are synthetic.
Errors and limits
Pentest findings depend on tester skill; changes of tester could affect counts.
Exercises
- EX033 — Classifier agreement. How many findings did classifiers agree on? Rubric: 3 = 46 of 48; 2 = one; 1 = vague; 0 = none.
- EX034 — Per-release figures. Give findings per release before and after. Rubric: 3 = 2.83 and 0.50; 2 = one; 1 = vague; 0 = none.
- EX035 — Rate ratio. What is the rate ratio? Rubric: 3 = 0.14 (0.415 to 0.058); 2 = partial; 1 = vague; 0 = none.
- EX036 — Change size. Why does change size strengthen the result? Rubric: 3 = after period changed more code; 2 = partial; 1 = vague; 0 = none.
- EX037 — Caution. What does Sofia avoid claiming? Rubric: 3 = 86% reduction caused by tests; 2 = partial; 1 = vague; 0 = none.
- EX038 — Product effect. What side effect did a developer describe? Rubric: 3 = matrix forced explicit role decisions; 2 = partial; 1 = vague; 0 = none.
Synthesis
Normalise, interpret small samples with care and let practitioners explain the numbers.
Continue studying — chapter 06 of 08
← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →