Chapter 19 — Scope and theory of authorized offensive security
Lesson map
Lesson 19 published. Map, example, EX055.
flowchart TD
N0["Learning outcomes"]
N1["19.1 SP 800-115 types"]
N2["19.2 PTES"]
N3["19.3 Threat-led"]
N4["19.4 Theory of change"]
N5["19.6 Platform role"]
N6["19.7 Stating what each test type cannot claim"]
N7["19.8 Approved sentences for reports"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX055"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["19.1 SP 800-115 types"]
N2["19.2 PTES"]
N3["19.3 Threat-led"]
N4["19.4 Theory of change"]
N5["19.6 Platform role"]
N6["19.7 Stating what each test type cannot claim"]
N7["19.8 Approved sentences for reports"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX055"]
[Learning outcomes]
|
v
[19.1 SP 800-115 types]
|
v
[19.2 PTES]
|
v
[19.3 Threat-led]
|
v
[19.4 Theory of change]
|
v
[19.6 Platform role]
|
v
[19.7 Stating what each test type cannot claim]
|
v
[19.8 Approved sentences for reports]
|
v
[Worked example] --> [Exercise EX055]
Learning outcomes
You will (1) compare assessment types in SP 800-115; (2) map PTES stages; (3) situate threat-led tests.
19.1 SP 800-115 types
Reviews, vulnerability assessments, red teaming — different depth and permission models.
NIST SP 800-115 distinguishes several kinds of security testing. A review examines designs, configurations or code without attacking a live system. A vulnerability assessment looks for known weaknesses with tools and light manual checks, usually with broad permission and limited stealth. Red teaming emulates an adversary with specific goals, often under tighter rules about detection and impact. Confusing the names leads to wrong expectations. A quarterly web assessment cannot answer whether a determined attacker would reach the ledger unnoticed, and a threat-led exercise cannot replace regular regression checks on the customer portal.
19.2 PTES
Pre-engagement through reporting as shared stage names.
The Penetration Testing Execution Standard names stages that most authorized tests share: pre-engagement, intelligence gathering, threat modelling, vulnerability analysis, exploitation, post-exploitation and reporting. TRB uses those names so that planners, testers and clients mean the same thing by each stage. Pre-engagement is where scope and rules are signed. Reporting is where findings are written for owners, not where new testing happens. Using shared stage names also makes gaps obvious. An engagement that jumps from intelligence gathering to reporting without analysis has skipped the work that turns observations into findings.
19.3 Threat-led
TIBER-EU and CBEST use intelligence to shape scenarios for financial entities; TRB studies them even when running lighter tests.
Threat-led exercises start from who is likely to attack this kind of institution and what they usually want, then build scenarios around that picture. TIBER-EU and CBEST are frameworks banks use for such exercises under regulatory oversight. Ordinary TRB web tests are lighter and more frequent, but the threat-led idea still shapes them. Scope meetings ask which techniques current threat reporting highlights for retail banks, and the method cards for the quarter favour those techniques. Studying the frameworks also keeps the team honest about what a light test cannot claim.
19.4 Theory of change
Testing informs risk decisions; it does not create security by itself.
A successful engagement produces evidence that a risk owner can use: a weakness exists, it is reachable under stated conditions, and fixing it would remove that path. Security improves only if someone then decides to fix it, accept it with a compensating control or change the system so the path no longer matters. Testing that produces findings nobody owns is expensive journalism. TrackScope therefore requires an owner and a due date before a finding can leave draft, and PulseMap tracks age so that informed-but-ignored risks stay visible.
19.5 Assurance limits
A clean test is time-bounded assurance, not a warranty.
A test that finds nothing serious shows that, under the methods used, against the systems in scope, during that window, the testers did not demonstrate a serious weakness. It does not show that no weakness exists, that none will appear next week when a new feature ships, or that a different adversary with different methods would fail. Reports state this limit explicitly. TRB forbids phrases that turn a clean test into a guarantee, because those phrases become liabilities when the next incident happens.
19.6 Platform role
TrackScope encodes theory as executable deny rules.
Ideas about authorization, scope and fail-closed behaviour only protect the bank when they run as code. TrackScope refuses to start a test without a signed scope file, refuses targets outside that file, refuses methods the rules mark as forbidden and stops when its policy cannot be loaded. Encoding the theory means a tired tester at midnight gets the same refusal as a careful one at noon. It also means that improvements to the theory, such as a new exclusion, become improvements to every engagement the moment the scope file is updated.
19.7 Stating what each test type cannot claim
The assessment catalogue includes what each type cannot claim. Annual web tests cannot claim threat-led coverage; threat-led exercises cannot claim quarterly regression coverage. Sam quotes the cannot-claim line in kickoffs.
Clients and internal sponsors naturally stretch what they bought. After a clean web test someone asks whether the bank is safe from nation-state actors. After a threat-led exercise someone asks why a simple regression bug reached production. The cannot-claim line, read aloud at kickoff and printed in the report, answers both before they become arguments. It also sets the catalogue's value. Buyers know what they are and are not purchasing.
19.8 Approved sentences for reports
Approved sentences for reports include "time-bounded assurance" and forbid "guaranteed secure". Sam's linter flags forbidden phrases.
Writers under deadline reach for strong language. A library of approved sentences gives them strong-enough language that legal and the CISO have already accepted. Forbidden phrases are blocked in the report pipeline with a message that suggests a replacement. The linter is intentionally blunt. It is easier to argue with a suggested replacement than to notice a dangerous phrase in a long draft at midnight.
19.9 Answering questionnaires from the catalogue
Questionnaire answers cite the catalogue. Sam stops colleagues from answering "we pentest everything" when SWIFT is excluded.
Insurance and vendor questionnaires ask broad yes-or-no questions that do not match how testing actually works. Answering from memory produces overclaims. Answering from the catalogue produces sentences such as: quarterly authorized web assessments of internet-facing applications, threat-led exercise last completed in the stated year, payment gateway excluded from ordinary tests. Those sentences are defensible in a claim dispute. "We pentest everything" is not.
19.10 An honest answer to an insurer
An insurer questionnaire says "Do you conduct red teaming?". Sam answers with the catalogue: quarterly web tests yes; full threat-led last done 20XX; SWIFT excluded from ordinary tests. Checkbox honesty beats swagger.
The form wanted a yes or no. The honest answer is a short paragraph, which the insurer accepted after one clarification call. A checkbox yes would have implied a current, comprehensive red-team programme that TRB did not have that year. When an incident later involved a payment-adjacent system, the written answer showed that ordinary tests had never claimed to cover it, which mattered more than looking assertive on the form.
19.11 Replacing a forbidden phrase
Assurance linter failed a report for "fully secure after remediation". Sam's replacement: "residual risk accepted for F3 until 1 December with compensating jump-only admin".
The forbidden phrase promised a state no test can prove. The replacement names the finding, the decision, the date and the compensating control. An owner can act on it, an auditor can check it and a later incident can be compared with it. Writing the specific sentence took longer than the slogan, and that is the point. Precision is the product.
19.12 Reading cannot-claim lines aloud
Catalogue cannot-claim lines are read aloud at kickoff; mumbling is corrected.
Reading aloud forces every person in the room to hear the limit, including sponsors who skip the written pack. Mumbling is corrected because a half-heard limit is how overclaims start. The practice feels slightly formal the first few times and then becomes normal. Kickoffs that skip it tend to produce the same clarifying arguments a week later, so the team treats the reading as part of starting the engagement, not as optional theatre.
19.13 Restarting a mumbled reading
Kickoff mumbling earns a restart of the cannot-claim read. Clarity over comfort.
A restart is mildly embarrassing and therefore effective. The reader slows down, the room pays attention and the limit is actually heard. Sam keeps the tone light. The goal is shared understanding, not punishment. After a few restarts early in the year, mumbled readings became rare, which is the outcome the rule exists to produce.
The restart also gives late arrivals a second chance to hear the limit. Kickoffs often start while someone is still joining the call, and a mumbled first reading is the one they would have caught. Waiting for a clean reading costs less than explaining the same boundary again mid-engagement.
19.14 Tracking mumbled kickoffs
Cannot-claim read restarts are logged as a fun metric of mumbling.
The log is a light-hearted count on the team wiki, not a performance score. A rising count would suggest that kickoffs are rushed or that new staff have not seen the practice. A falling count suggests the habit has stuck. Treating the metric as fun keeps it from becoming something people game by reading too quickly to avoid a restart. The useful signal is the trend across quarters, not any single kickoff.
Worked example (synthetic TRB)
Sam's lecture note: "CBEST-style scenarios need intel + ROE + detection goals; our quarterly web test is narrower and still valuable."
Synthetic case — Riverstone Transit Bank (TRB). The bank, Sam Ito, TrackScope/FindingsDesk/PulseMap and all figures are invented for teaching.
Errors and limits
This book grants no authority to test real banks or payment systems. Labs use documentation-range addresses and synthetic data only; exploit code is never provided.
Exercises
- EX055 — Stage. Where does PTES place reporting? Rubric: 3 = final stage after technical work; 2 = early; 1 = never; 0 = none.
- EX056 — Threat-led pair. Name the EU and UK threat-led frameworks cited. Rubric: 3 = TIBER-EU and CBEST; 2 = one; 1 = wrong; 0 = none.
- EX057 — Warranty. Why is a clean test not a warranty? Rubric: 3 = time-bounded / coverage limits / change; 2 = partial; 1 = it is; 0 = none.
Synthesis
Theory ties assessment types to permissions, stages and humble assurance claims.
Continue studying — chapter 19 of 40
← Chapter 18 (draft) · Course chapter index · Chapter 20 (draft) →