Chapter 33 — Research methods II | Bachelor's Authorized offensive security and red team

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 33 — Research methods II

Lesson map

Lesson 33 published. Map, example, EX097.

flowchart TD
  N0["Learning outcomes"]
  N1["33.1 Analysis"]
  N2["33.2 Validity"]
  N3["33.3 Adaptive evaluation"]
  N4["33.5 Data sharing"]
  N5["33.6 Ethics revisit"]
  N6["33.7 Three validity threats"]
  N7["33.8 Adaptation log"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX097"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["33.1 Analysis"]
  N2["33.2 Validity"]
  N3["33.3 Adaptive evaluation"]
  N4["33.5 Data sharing"]
  N5["33.6 Ethics revisit"]
  N6["33.7 Three validity threats"]
  N7["33.8 Adaptation log"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX097"]

[Learning outcomes]
    |
    v
[33.1 Analysis]
    |
    v
[33.2 Validity]
    |
    v
[33.3 Adaptive evaluation]
    |
    v
[33.5 Data sharing]
    |
    v
[33.6 Ethics revisit]
    |
    v
[33.7 Three validity threats]
    |
    v
[33.8 Adaptation log]
    |
    v
[Worked example] --> [Exercise EX097]

Learning outcomes

You will (1) analyse purple-team results; (2) respect threats to validity; (3) report uncertainty.

33.1 Analysis

Use intervals; avoid p-hacking by switching metrics post hoc.

Analysis plans name the metric and the interval before data collection starts. Sam rejects study drafts that keep three spare metrics in the drawer for a prettier slide. Intervals communicate uncertainty; point estimates alone invite false precision. Post-hoc switches are allowed only as exploratory notes clearly labeled as such. The lab's report template has a locked metrics box that cannot be edited after data land. Precommitment is how classroom studies stay honest when results disappoint.

33.2 Validity

Staging, alert fatigue, shared accounts threaten generalisation.

Validity sections must name threats that actually apply to TRB-like environments. Staging that lacks production noise, analysts drowning in alerts and shared admin accounts all warp what a detection score means. Sam's rubric docks reports that only mention sample size. Threats are paired with a mitigation or an explicit limit on claims. Readers should know how far the result can travel. Overclaiming from a tidy lab is a validity failure, not a marketing win.

33.3 Adaptive evaluation

Adversaries adapt; static test sets age.

Evaluation that never refreshes becomes a museum of last year's tricks. Sam logs when attackers or internal systems change enough to stale a static test. Adaptive evaluation schedules refresh triggers, not only annual reviews. A perfect score on a frozen set can hide a blind spot that opened last month. The research group treats staleness as a first-class risk. Living opponents require living tests if assurance language is going to mean anything. Refresh triggers belong in the plan, not in a guilty afterthought.

33.4 Disclosure of limits

Every study lists limits in the abstract.

Limits buried on page twelve might as well not exist for busy readers. Sam requires at least three limits in the abstract: environment, sample and adversary adaptation. The PDF build fails if the list is shorter. Naming limits early is a feature that builds trust with skeptical reviewers. Students who hide limits to look strong lose points for scientific manners. Abstracts that admit weakness teach better judgment than abstracts that boast. Putting limits up front trains readers to trust the rest of the page.

33.5 Data sharing

Synthetic datasets preferred for sharing.

Raw TRB logs do not leave the bank for a classroom paper. Sam's default is a synthetic dataset with documented generators and known ground truth. When real traces are essential, sharing stays inside approved research environments with access logs. Synthetic does not mean toy; generators encode the skew the study cares about. Reviewers receive generator notes so they can judge realism. Preferring synthetic data keeps publication ambitions from becoming data leaks. Generator notes let outsiders judge realism without touching bank logs.

33.6 Ethics revisit

Re-check approvals if methods expand mid-study.

Scope creep in methods can outrun the original ethics approval. Sam pauses collection when a new technique appears and files an amendment before continuing. Students sometimes treat approval as a one-time gate; the course treats it as a living bound. Expanding into staff-affecting steps reopens the HR path as well. Unapproved expansion voids the study for course credit. Ethics that cannot be revisited is decoration, not control. Amendments are cheaper than discoveries that ethics never authorised.

33.7 Three validity threats

Validity sections list at least three threats every time: environment, sample size, and adversary adaptation. Sam's template will not compile the report PDF if the list has fewer than three.

The three-threat minimum stops empty validity theatre. Environment covers staging versus production differences students love to ignore. Sample size covers the quiet n that inflated a percentage. Adversary adaptation covers the fact that last quarter's payload may be loud now. Mechanical enforcement keeps tired writers from skipping the hard paragraph. Templates that refuse to compile soft sections raise the scientific floor.

33.8 Adaptation log

When attackers or systems change, Sam logs which static tests are stale. The log feeds the next trial's design.

The adaptation log is a dated list of what changed and which tests it wounds. Sam reviews it before proposing a new trial so the design does not pretend the world froze. Entries cite tickets or intel notes rather than vibes. Tests marked stale cannot support assurance claims until refreshed. The log is short enough to read at the start of planning meetings. Design that ignores the log repeats last year's blind spots with new confidence.

33.9 Stale-test date

Tests carry a review-by date. Past-due tests cannot support assurance claims until refreshed.

Every packaged test has a review-by date beside its name. Past-due tests may still run for training, but they cannot be cited in assurance language. Sam's assurance checklist asks whether any red-dated tests were cited and requires a no. Dates force an adult conversation about refresh funding. Eternal tests are a fantasy that regulated banks cannot afford. Expiry dates on tests are how research hygiene becomes operational hygiene.

33.10 Traffic-light test ages

Stale-test dates turn orange at thirty days and red at ninety. Red tests cannot be cited in assurance statements until re-run. Sam likes traffic lights more than paragraphs of policy.

Colour on the dashboard beats a policy PDF nobody opens. Orange means schedule a refresh; red means citation is blocked. Sam showed leadership the lights once and funding for refresh work became easier to explain. Students learn the same lights in the lab so the habit transfers. Policy paragraphs remain for auditors; operators steer by colour. Visible aging turns abstract freshness rules into daily decisions.

33.11 Orange schedule, red block

Orange tests get scheduled; red tests block citations. Sam's assurance checklist literally asks "any red tests cited?" with a no-box required.

Scheduling is the constructive response to orange; blocking is the hard response to red. The checklist's no-box is a forced pause before someone pastes an old score into a board pack. Sam has rejected packs that left the box blank. Automation later queries the same state the human checklist asks about. Humans and bots agreeing on red is the goal. Forced questions catch the citation that hope would have smuggled through.

33.12 Traffic-light API in builds

Traffic-light API is queried by the report build; red citations fail the build.

Report builds call the traffic-light service and fail closed on red citations. Authors see which test id tripped the build and must refresh or remove the claim. Sam prefers a failing build to a polite warning that busy people dismiss. The API is simple on purpose: status by test id, nothing more. Offline builds that cannot reach the API also fail rather than assume green. Failing closed keeps stale assurance language out of finished PDFs.

33.13 Offline builds fail closed

Report builds call the traffic-light API; offline builds fail closed.

Convenience builds on laptops without network used to skip the check. That skip was a hole large enough to drive a stale claim through. Sam changed the default so offline equals fail until someone deliberately uses an approved offline bundle with pinned statuses. Pinned bundles expire quickly. The annoyance is cheaper than a board pack citing a red test. Network-optional honesty is still honesty when the fallback is denial. Pinned offline bundles expire fast so convenience cannot become a back door.

33.14 Ban on prove without threshold

Research office hours ban the word "prove" unless a pre-registered threshold is cited. Sam rings a small bell. The room self-corrects by week three.

Casual prove-talk inflates what a classroom study can carry. The bell is slightly silly and extremely effective at interrupting the habit. Speakers must cite the pre-registered threshold or rephrase as supports or consistent with. In the first week the bell rings several times an hour, mostly at students describing the awareness-mail trial as proof that the content pack failed. Language discipline is part of research methods, not pedantry. Small rituals can police big epistemic bad habits without lengthy lectures.

Worked example (synthetic TRB)

Sam reports T1190 GAP with n=2 and refuses to claim "zero detection forever".

Synthetic case — Riverstone Transit Bank (TRB). The bank, Sam Ito, TrackScope/FindingsDesk/PulseMap and all figures are invented for teaching.

Errors and limits

This book grants no authority to test real banks or payment systems. Labs use documentation-range addresses and synthetic data only; exploit code is never provided.

Exercises

EX097 — Post hoc. Why not change the success metric after seeing data? Rubric: 3 = p-hacking / biased claim; 2 = partial; 1 = fine; 0 = none.
EX098 — n=2. How should Sam talk about n=2 GAP? Rubric: 3 = honest small-n uncertainty; 2 = partial; 1 = proof of zero forever; 0 = none.
EX099 — Adaptive. Why do static test sets age? Rubric: 3 = adversaries and systems change; 2 = partial; 1 = they do not; 0 = none.

Synthesis

Research honesty is intervals, listed limits and stable metrics.

Continue studying — chapter 33 of 40

← Chapter 32 (draft) · Course chapter index · Chapter 34 (draft) →