Chapter 6 — Peer critique and revision | Doctoral research Authorized offensive security and red team

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 06 published. Map, example, EX033.

flowchart TD
  N0["Learning outcomes"]
  N1["6.1 Who critiques"]
  N2["6.2 Major vs minor"]
  N3["6.3 Revision without switching"]
  N4["6.4 Threat model of the study"]
  N5["6.4b Practitioner translation"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX033"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["6.1 Who critiques"]
  N2["6.2 Major vs minor"]
  N3["6.3 Revision without switching"]
  N4["6.4 Threat model of the study"]
  N5["6.4b Practitioner translation"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX033"]

[Learning outcomes]
    |
    v
[6.1 Who critiques]
    |
    v
[6.2 Major vs minor]
    |
    v
[6.3 Revision without switching]
    |
    v
[6.4 Threat model of the study]
    |
    v
[6.4b Practitioner translation]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX033]

Learning outcomes

You will (1) solicit critique from methodology and practice experts; (2) separate major from minor issues; (3) revise without outcome-switching; (4) update the threat model of your own study; and (5) record a response-to-reviewers trail.

6.1 Who critiques

Invite: a red-team practitioner (ecological validity), a measurement researcher (statistics), an ethicist or counsel (authorization), and a blue-team lead (whether your reports would help defenders). Share redacted materials.

For Elena, each reviewer tests a different weak point. The practitioner asks whether NHUSTL's isolated 10.20.0.0/16 and 192.168.50.0/24 ranges resemble the networks where scope mistakes actually happen. The measurement researcher checks whether out-of-scope probes per engagement-hour, compared at 20 engagements per arm, supports the interval she reports. Counsel reads the authorization file, and the blue-team lead judges whether a deny-reason log would help defenders. Redacted materials still include the pre-registration, so each reviewer can see what she promised before the data arrived.

6.2 Major vs minor

Major: the measure does not match the claim; independence is compromised; ethics gap; results driven by one operator. Minor: wording, figure layout, related-work gaps. Major issues change conclusions or require new data.

Elena sorts every comment into one of these two lists before replying. The practitioner's point that NHUSTL's topology is too clean is major, because it bounds where the drop from 0.17 to 0.03 out-of-scope probes per hour can be claimed. A request to relabel a figure axis is minor. Sorting first stops her spending a week on presentation while a validity question waits, and the response trail shows reviewers that their major points received the larger share of revision effort.

6.3 Revision without switching

You may clarify, add sensitivity analyses, or collect new data under an amended pre-registration. You may not redefine success after seeing results. If the primary fails, say so and discuss why.

In Elena's case the pre-registered primary measure, out-of-scope probes per engagement-hour, stays fixed. She can add a sensitivity analysis that drops engagements aborted for infrastructure failure, because that exclusion was already named in the pre-registration. She cannot swap in time-to-stop as the headline if it happens to look stronger; it remains a secondary measure. The amendment log records each change with its date, so an examiner can compare the revised thesis against the plan filed before the main study.

6.4 Threat model of the study

Attackers of your study include: operators who game the checker; lab infra that drifts; validators who share incentives. Document residual risks to validity.

For NHUSTL, gaming the checker would mean operators splitting a probe across permitted ports or proposing actions the predicate cannot express, so that the checker records no violation while the behaviour still drifts. Lab drift would include a range renumbered mid-study, which would make an allowed CIDR silently cover new hosts. Validators who share Elena's supervisor share her incentive to find an effect. Each threat gets a recorded mitigation or is listed as a residual limit in the thesis.

"""Password policy strength from synthetic samples (no cracking)."""
import math, string
samples = [
    "password",
    "P@ssw0rd",
    "correct-horse-battery",
    "Xk9!mQ2#vL7$",
    "aaaaaaaa",
]

def charset_size(s):
    n = 0
    if any(c.islower() for c in s): n += 26
    if any(c.isupper() for c in s): n += 26
    if any(c.isdigit() for c in s): n += 10
    if any(c in string.punctuation+"-" for c in s): n += 33
    return max(n, 1)

def entropy_bits(s):
    # naive: length * log2(charset); flags common patterns
    bits = len(s) * math.log2(charset_size(s))
    flags = []
    if s.lower() in {"password","admin","welcome","letmein"}: flags.append("common")
    if len(set(s)) == 1: flags.append("single-char")
    if s.isalpha() and s.istitle() and any(c.isdigit() for c in s): flags.append("word+digit")
    return bits, flags

for s in samples:
    b, f = entropy_bits(s)
    grade = "weak" if b < 40 or f else ("ok" if b < 60 else "strong")
    print(f"sample={s!r:28} bits={b:5.1f} grade={grade:6} flags={','.join(f) or '-'}")

Expected output:

sample='password'                   bits= 37.6 grade=weak   flags=common
sample='P@ssw0rd'                   bits= 52.6 grade=ok     flags=-
sample='correct-horse-battery'      bits=123.5 grade=strong flags=-
sample='Xk9!mQ2#vL7$'               bits= 78.8 grade=strong flags=-
sample='aaaaaaaa'                   bits= 37.6 grade=weak   flags=single-char

6.4b Practitioner translation

After academic revision, write a one-page practitioner brief: what to configure, what to measure on the next authorized engagement, and what not to claim. If practitioners cannot use the brief, the contribution's path to impact is incomplete.

Elena's brief states the checker defaults, the two measures to watch on the next authorized engagement, and the refused claims: residual risk from coverage percentages, and replacement of threat-led programmes. She asks one NHUSTL operator to follow the brief cold and notes where they stall.

6.5 Worked example (synthetic NHUSTL case)

A practitioner argues that NHUSTL's topology is too clean versus enterprise networks. Elena adds a limitations subsection and a planned external replication, without changing the primary claim. A statistician asks for operator-level plots; she adds them as exploratory. The entropy lab is used in a teaching appendix on why "complexity scores" without context mislead — analogous to empty security scores.

The limitation and planned external replication answer the practitioner without moving the goalposts. The entropy lab's lesson — that bit counts without context grade the common pattern P@ssw0rd as ok at 52.6 bits — parallels empty security scores, and sits in a teaching appendix rather than in the primary claim.

6.6 Errors and limits

Ignoring major critique.
Quietly changing the primary measure.
Defensive responses without evidence.

Each error damages honesty. Ignoring major critique leaves validity questions for examiners to find. Quietly changing the primary measure moves the goalposts after seeing results. Defensive responses without evidence do not convince reviewers or examiners. Check the response trail for all three before the next revision. Elena's response trail shows major issues first, with evidence for each reply. When a reviewer questioned the false-stop rate, she answered with the pilot logs and the typo fix rather than an argument.

Exercises

EX033 — Reviewers. Name four reviewer roles and one question each. Rubric: 3 = four apt; 2 = three; 1 = one; 0 = none.
EX034 — Major issue. Give an example major issue for Elena and a fix. Rubric: 3 = issue and honest fix; 2 = issue; 1 = cosmetic; 0 = none.
EX035 — No switching. What may she change after results? Rubric: 3 = clarify/sensitivity/new data with amendment; 2 = partial; 1 = redefine success; 0 = none.
EX036 — Validity threat. List three threats to validity. Rubric: 3 = three with mitigations; 2 = two; 1 = one; 0 = none.
EX037 — Entropy lab. How does the lab support scepticism of single scores? Rubric: 3 = bits without context mislead; analogy; 2 = partial; 1 = vague; 0 = wrong.
EX038 — Response trail. Outline a response-to-reviewers table. Rubric: 3 = comment, response, change location; 2 = two; 1 = one; 0 = none.

Synthesis

Critique improves honesty. Revise what the evidence supports; never silently move the goalposts.

Continue studying — chapter 06 of 08

← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →