Chapter 1 — Advanced research integrity and methods
Lesson map
Lesson 01 published. Map, example, EX001.
flowchart TD
N0["Learning outcomes"]
N1["1.1 Integrity as a research method"]
N2["1.2 Authorization, scope and stop conditions"]
N3["1.3 Measures that matter"]
N4["1.4 Dual-use discipline"]
N5["1.5 Evidence trail"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX001"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["1.1 Integrity as a research method"]
N2["1.2 Authorization, scope and stop conditions"]
N3["1.3 Measures that matter"]
N4["1.4 Dual-use discipline"]
N5["1.5 Evidence trail"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX001"]
[Learning outcomes]
|
v
[1.1 Integrity as a research method]
|
v
[1.2 Authorization, scope and stop conditions]
|
v
[1.3 Measures that matter]
|
v
[1.4 Dual-use discipline]
|
v
[1.5 Evidence trail]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX001]
Learning outcomes
You will (1) state the integrity rules that govern authorized offensive-security research; (2) design studies that cannot harm people or out-of-scope systems; (3) choose measures that reflect effects, not theatre; (4) handle dual-use knowledge responsibly; and (5) keep an evidence trail others can audit.
1.1 Integrity as a research method
In this field, a failed integrity control is not a paperwork issue — it is an out-of-scope probe, a leaked credential, or a denial of service against a hospital. Integrity means: written authorization before any interaction with a target; a scope that a third party can check; stop conditions that are rehearsed; synthetic or consented data; and records that show what was done and why. NIST SP 800-115 frames assessment as a planned activity with rules; PTES places scoping and intelligence before any testing. Your doctorate inherits those habits and adds the standards of research: pre-registration of measures, replication and honest limits.
1.2 Authorization, scope and stop conditions
Authorization names the owner, the systems, the dates, the allowed actions and the data rules. Scope is the set of hosts, ports, identities and actions that are in. Stop conditions — for example, any real personal data, any out-of-scope host, any availability impact — pause work and escalate. A machine-checkable scope (CIDR, port and action lists) makes violations detectable rather than arguable.
"""Check proposed engagement targets against a written scope (synthetic)."""
SCOPE_CIDRS = [("10.20.0.0", 16), ("192.168.50.0", 24)] # inclusive synthetic ranges
SCOPE_PORTS = {22, 80, 443, 8080}
DENY_HOSTS = {"10.20.0.1", "192.168.50.1"} # out-of-scope infra
def ip_int(s):
a,b,c,d = map(int, s.split("."))
return (a<<24)|(b<<16)|(c<<8)|d
def in_cidr(ip, net, bits):
mask = (0xFFFFFFFF << (32-bits)) & 0xFFFFFFFF
return (ip_int(ip) & mask) == (ip_int(net) & mask)
def check(host, port):
reasons = []
if host in DENY_HOSTS:
reasons.append("deny-list")
if not any(in_cidr(host, n, b) for n,b in SCOPE_CIDRS):
reasons.append("outside-cidr")
if port not in SCOPE_PORTS:
reasons.append("port-not-in-scope")
return ("ALLOW" if not reasons else "DENY"), reasons
tests = [
("10.20.1.10", 443),
("10.20.0.1", 443),
("192.168.50.20", 22),
("192.168.50.20", 3389),
("8.8.8.8", 80),
("192.168.1.10", 80),
]
for h,p in tests:
v, r = check(h,p)
print(f"target={h}:{p} decision={v} reasons={','.join(r) or '-'}")
Expected output:
target=10.20.1.10:443 decision=ALLOW reasons=-
target=10.20.0.1:443 decision=DENY reasons=deny-list
target=192.168.50.20:22 decision=ALLOW reasons=-
target=192.168.50.20:3389 decision=DENY reasons=port-not-in-scope
target=8.8.8.8:80 decision=DENY reasons=outside-cidr
target=192.168.1.10:80 decision=DENY reasons=outside-cidr
Elena's lab uses the ranges and ports in the worked example: probes aimed at 8.8.8.8 or at port 3389 are denied before any packet leaves. Write the stop conditions into the runbook and rehearse them, so operators know what to do when one fires.
1.3 Measures that matter
Count effects: out-of-scope probes, confirmed findings with evidence, time-to-stop after a violation, remediation verification. Avoid vanity counts such as "number of tools run". Report rates with intervals. When you compare methods, change one factor at a time and hold the environment fixed.
"""Lab S: reporting attack-success rates honestly: Wilson intervals,
seed variation and a paired bootstrap. Synthetic trial outcomes only."""
import math, random
def wilson(k, n, z=1.96):
p = k/n; d = 1 + z*z/n
c = (p + z*z/(2*n))/d; h = z*math.sqrt(p*(1-p)/n + z*z/(4*n*n))/d
return c - h, c + h
for k, n in ((3, 20), (30, 200), (300, 2000)):
lo, hi = wilson(k, n); print(f"successes={k}/{n} rate={k/n:.3f} wilson95=[{lo:.3f}, {hi:.3f}]")
# five seeds of a synthetic defence evaluation: per-seed attack success rate
rates = []
for seed in range(5):
r = random.Random(seed); rates.append(sum(r.random() < 0.12 for _ in range(200))/200)
m = sum(rates)/5; sd = math.sqrt(sum((x-m)**2 for x in rates)/4)
print("per-seed rates:", " ".join(f"{x:.3f}" for x in rates), f"mean={m:.3f} sd={sd:.3f}")
# paired bootstrap: baseline vs defence on the same 300 synthetic trials
r = random.Random(42)
base = [r.random() < 0.30 for _ in range(300)]
defn = [b and r.random() < 0.6 for b in base] # defence blocks ~40% of successes
diffs = []
for _ in range(2000):
idx = [r.randrange(300) for _ in range(300)]
diffs.append(sum(base[i] for i in idx)/300 - sum(defn[i] for i in idx)/300)
diffs.sort()
print(f"baseline_asr={sum(base)/300:.3f} defended_asr={sum(defn)/300:.3f} "
f"reduction_ci95=[{diffs[50]:.3f}, {diffs[1949]:.3f}]")
Expected output:
successes=3/20 rate=0.150 wilson95=[0.052, 0.360]
successes=30/200 rate=0.150 wilson95=[0.107, 0.206]
successes=300/2000 rate=0.150 wilson95=[0.135, 0.166]
per-seed rates: 0.110 0.130 0.135 0.105 0.145 mean=0.125 sd=0.017
baseline_asr=0.350 defended_asr=0.190 reduction_ci95=[0.120, 0.203]
The stats lab shows why counts matter: 3 successes in 20 and 300 in 2,000 both give a 15% rate, but the Wilson intervals are about 5–36% and 13.5–16.6%. Only the larger sample supports a precise claim. Elena's primary measure is out-of-scope probes per engagement-hour, with an interval.
1.4 Dual-use discipline
Your thesis will discuss techniques that could be misused. Publish methods and measurement, not weaponised procedures. Prefer synthetic targets and open lab topologies. Route any real weakness found during research through coordinated disclosure. Herzog's OSSTMM has been influential, yet Albrecht and Jensen (arXiv:2010.06377) show why a single "security quantity" is a poor research measure — prefer concrete, contextual metrics.
Elena's protocol states that she will publish predicates, rates and scoring code, not exploit chains, and that any real issue found on institutional systems goes through coordinated disclosure. Her measures are concrete — probe rates and time-to-stop — not a single security score.
1.5 Evidence trail
Keep: authorization, ROE, scope versions, tool versions, raw logs (redacted), findings with evidence hashes, and a change log. An auditor should be able to answer "what was tried, against what, when, under which permission?"
Elena's trail also records which scope version was active when each probe was logged, so a later dispute about what was in can be settled from the record rather than from memory. Pin tool versions beside the logs. Store the trail where only the study team and supervisors can read it.
1.6 Worked example (synthetic NHUSTL case)
Elena's first integrity plan: all studies run on NHUSTL's isolated 10.20.0.0/16 and 192.168.50.0/24 ranges; ports limited to 22, 80, 443, 8080; deny-list includes 10.20.0.1. The scope lab shows how a proposed probe of 8.8.8.8:80 or 192.168.50.20:3389 is denied before packets leave. She pre-registers two measures: out-of-scope probe rate and minutes from violation note to stop.
Those two measures become her primary and secondary outcomes. The deny decisions in the scope lab are the first evidence that the checker works before any engagement starts. Both appear in her integrity plan before the first engagement.
1.7 Errors and limits
- Verbal authorization.
- Measures that count activity instead of effects.
- Publishing working exploit recipes.
Each error damages integrity from the start. Verbal authorization cannot be checked later and does not protect anyone if a probe causes harm. Counting tools run or packets sent says nothing about whether scope held. Publishing working exploit recipes turns research into an attack resource. Check your integrity plan for all three before any engagement starts. Elena's plan names written authorization, effect measures and dual-use limits before any engagement starts.
Exercises
- EX001 — Authorization. Draft the authorization fields for Elena's study. Rubric: 3 = owner, systems, dates, actions, data, contacts; 2 = four; 1 = two; 0 = none.
- EX002 — Scope lab. Explain two DENY decisions from the lab. Rubric: 3 = both with matching reasons; 2 = one; 1 = vague; 0 = wrong.
- EX003 — Stop condition. Write three stop conditions with escalation. Rubric: 3 = three with who/how; 2 = two; 1 = one; 0 = none.
- EX004 — Measure. Propose a primary measure for comparing prose ROE vs machine-checkable scope. Rubric: 3 = effect measure, interval plan, confounders; 2 = measure only; 1 = vanity count; 0 = none.
- EX005 — Stats lab. Why is 300/2000 more informative than 3/20 at the same rate? Rubric: 3 = interval width; 2 = partial; 1 = vague; 0 = wrong.
- EX006 — Dual-use. How will you publish without enabling misuse? Rubric: 3 = methods/measurement, no payloads, synthetic targets, CVD path; 2 = three; 1 = one; 0 = none.
- EX007 — Evidence. List six artefacts in the evidence trail. Rubric: 3 = six apt; 2 = four; 1 = two; 0 = none.
Synthesis
Doctoral integrity in authorized offensive security means permission, checkable scope, effect measures, dual-use restraint and an auditable trail — before any claim of contribution.