Chapter 5 — Tools and interpretation
Lesson map
Lesson 05 published. Map, example, EX021.
flowchart TD
N0["Learning outcomes"]
N1["5.1 Tools assist"]
N2["5.2 Confidence floor"]
N3["5.3 False positives and negatives"]
N4["5.4 Versions and settings"]
N5["5.5 OWASP WSTG as a tool of thought"]
N6["5.6 Interpreting banners"]
N7["5.7 Base rates"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX021"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["5.1 Tools assist"]
N2["5.2 Confidence floor"]
N3["5.3 False positives and negatives"]
N4["5.4 Versions and settings"]
N5["5.5 OWASP WSTG as a tool of thought"]
N6["5.6 Interpreting banners"]
N7["5.7 Base rates"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX021"]
[Learning outcomes]
|
v
[5.1 Tools assist]
|
v
[5.2 Confidence floor]
|
v
[5.3 False positives and negatives]
|
v
[5.4 Versions and settings]
|
v
[5.5 OWASP WSTG as a tool of thought]
|
v
[5.6 Interpreting banners]
|
v
[5.7 Base rates]
|
v
[Worked example] --> [Exercise EX021]
Learning outcomes
You will (1) treat scanners as assistants not judges; (2) filter findings by confidence and confirmation; (3) explain false positives and false negatives; (4) record tool versions in evidence.
5.1 Tools assist
Scanners and checklists speed discovery. They mislabel, miss context and cannot judge business impact. Leo reads every kept item: when lab_severity keeps three of five items, he still opens each kept item and asks whether HCU should act. Items he cannot explain in plain words about member impact are sent back for more validation or dropped. The two dropped items, dir-listing and trace-method, are noted in the method appendix so nobody wonders whether they were ignored. Scanners assist discovery; they do not judge member impact or write the recommendation.
5.2 Confidence floor
HCU drops unconfirmed items below a confidence floor so the report is not padded with noise.
"""Lab 5: filter a synthetic tool finding list, dropping noise below a confidence floor."""
findings = [("open-redirect", 0.9, True), ("dir-listing", 0.4, False),
("outdated-banner", 0.7, True), ("cookie-flag", 0.85, True), ("trace-method", 0.3, False)]
FLOOR = 0.6
kept = [(n, c) for n, c, confirmed in findings if c >= FLOOR and confirmed]
for n, c in kept:
print(f"KEEP {n:16} confidence={c}")
print(f"kept {len(kept)} of {len(findings)}; dropped {len(findings)-len(kept)}")
Run with python3 lab_severity.py. Verified output:
KEEP open-redirect confidence=0.9
KEEP outdated-banner confidence=0.7
KEEP cookie-flag confidence=0.85
kept 3 of 5; dropped 2
With FLOOR at 0.6, the lab keeps open-redirect (0.9), outdated-banner (0.7) and cookie-flag (0.85), and drops the rest. Dir-listing and trace-method fall below 0.6, so they stay in the raw output appendix rather than the findings list, and unconfirmed items below the floor never appear in executive slides. Raising the floor to 0.8 would drop the banner too. Leo states the floor in the method notes so readers know what was filtered.
5.3 False positives and negatives
A false positive wastes fixer time; a false negative leaves risk. Sampling and manual validation reduce both; neither reaches zero. Leo plans validation time from past false-positive rates and accepts that some risk may still be missed. If the last review's false positives clustered in one scanner check, that check gets extra validation time in ENG-440. When in doubt, he validates more rather than shipping a longer list. He tells the CISO the report contains validated items, not that every possible issue was found.
5.4 Versions and settings
Record tool name, version and profile with the evidence so a retest can use the same setup. Leo's evidence for the cookie-flag finding names the scanner, its version and the profile used, and the profile name is stored on ticket T-15, so the retest after app-sec's fix repeats the same check with the same scan settings. Without versions, a failed retest cannot tell whether the fix failed or the setup changed. Settings that change mid-engagement are logged with the time of the change, so the evidence still says which setup produced each result.
5.5 OWASP WSTG as a tool of thought
Even without a scanner, WSTG gives a structured list of web checks so coverage is visible. Leo uses a WSTG-derived checklist for online banking staging so he can show which checks ran and which were out of scope. Rows are marked ran, not run or out of scope, and each ran row points to its evidence file in the vault. Coverage that is only in his head cannot be reviewed or retested. Empty rows on the checklist are honest gaps; inventing coverage helps nobody when examiners ask what was actually tested.
5.6 Interpreting banners
An outdated banner is a clue, not a crisis. Leo checks whether the service is internet-facing, whether the version is truly vulnerable, and whether a WAF or access rule already limits exposure before he writes a high-severity finding. On HCU's staging tier, an outdated banner behind access rules may be low priority; the same banner on an internet-facing admin port would not. For the ENG-440 banner, Leo records that staging sits behind access rules, which is why it is not rated high. He writes that reasoning into the finding so severity is not a mysterious label divorced from member impact; context turns a clue into a proportional finding.
5.7 Base rates
If a scanner marks forty items and history shows thirty were false positives, Leo plans validation time accordingly: forty raw items with that history suggest roughly ten real findings. Base rates protect the schedule. He does not promise the CISO "forty findings" before validation, because that sets the wrong expectation and pressures the team to keep noise. Validation time is part of the engagement budget, not optional overtime, and Leo promises a validated set with a time box for confirmation.
5.8 Manual spot checks
For every five automated items kept, Leo manually validates at least the highest-severity one end-to-end with an authorized account. Spot checks catch systematic tool mistakes that a confidence score cannot, because those mistakes show up when a human repeats the path. In ENG-440 lab_severity keeps three items and open-redirect at 0.9 is the highest, so Leo walks that path first with an authorized account before the report is drafted. A failed walkthrough of open-redirect means the other kept items get the same human path before the report drafts.
Worked example (synthetic HCU)
Leo filters the synthetic list: open-redirect, outdated-banner and cookie-flag are kept; dir-listing and trace-method drop. He still manually validates the three kept items before they enter the report.
Synthetic case — Harborline Credit Union (HCU). The credit union, Leo Okonkwo and all figures are invented for teaching.
Errors and limits
Local policies and laws differ; follow HCU's authorization and your jurisdiction. This book never authorizes testing of real systems.
Exercises
- EX021 — Lab keep count. How many findings does the lab keep, and which are dropped? Rubric: 3 = 3 kept; dir-listing and trace-method dropped; 2 = count only; 1 = wrong; 0 = none.
- EX022 — Raise the floor. If FLOOR becomes 0.8, which items remain? Rubric: 3 = open-redirect and cookie-flag only; 2 = one; 1 = all three; 0 = none.
- EX023 — False positive. Give an HCU example of a false positive. Rubric: 3 = scanner flags issue that staging config shows is mitigated; 2 = partial; 1 = vague; 0 = none.
- EX024 — False negative. Give an HCU example of a false negative. Rubric: 3 = issue exists but tool misses (e.g. auth logic flaw); 2 = partial; 1 = vague; 0 = none.
- EX025 — Version note. Write the tool line Leo adds to evidence. Rubric: 3 = name, version, profile/settings; 2 = name only; 1 = vague; 0 = none.
Synthesis
Tools propose; humans dispose — filter, validate, record versions, then report.
Continue studying — chapter 05 of 12
← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →