Chapter 6 — Peer critique and revision

Lesson map

Lesson 06 published. Map, example, EX034.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["6.1 Critique as method"]
  N2["6.2 The red-team report on the Cascade Study synthet"]
  N3["6.3 Practical activity: boundary conditions, mismatc"]
  N4["6.4 Sensitivity analysis beyond one factor at a time"]
  N5["6.5 Negative results and narrowing"]
  N6["6.6 Dual-use review of the findings"]
  N7["6.7 Responding to reviewers"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX034"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["6.1 Critique as method"]
  N2["6.2 The red-team report on the Cascade Study synthet"]
  N3["6.3 Practical activity: boundary conditions, mismatc"]
  N4["6.4 Sensitivity analysis beyond one factor at a time"]
  N5["6.5 Negative results and narrowing"]
  N6["6.6 Dual-use review of the findings"]
  N7["6.7 Responding to reviewers"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX034"]

[Learning outcomes and connection]
    |
    v
[6.1 Critique as method]
    |
    v
[6.2 The red-team report on the Cascade Study synthet]
    |
    v
[6.3 Practical activity: boundary conditions, mismatc]
    |
    v
[6.4 Sensitivity analysis beyond one factor at a time]
    |
    v
[6.5 Negative results and narrowing]
    |
    v
[6.6 Dual-use review of the findings]
    |
    v
[6.7 Responding to reviewers]
    |
    v
[Worked example] --> [Exercise EX034]

Learning outcomes and connection

By the end of this chapter the learner will be able to:

1. Organize adversarial critique of one's own thesis — internal red-teaming, structured external review and adversarial collaboration — and record it.
2. Detect model–policy circularity and test it with deliberate model mismatch.
3. Map boundary conditions with sensitivity analysis and with many generated contexts, while recognizing that the generator is itself a model.
4. Distinguish a confirmatory revision from an exploratory one and label each.
5. Write negative and narrowed results so that they are useful to others.
6. Conduct a dual-use review of research findings and decide what to publish, generalize, delay or restrict.
7. Write a response to reviewers that changes claims rather than defending them.

Chapter 5 produced a mixed result: strong benefit over customer-first ordering, a trade-off that failed its margin, no benefit over critical-flag lists in one region and a large benefit in another. This chapter is where the thesis earns its claims by trying to destroy them.

6.1 Critique as method

Peer review at publication is too late and too shallow to be the main quality control of a doctoral project. Build critique into the project:

• Internal red team. Before submission, a colleague who did not work on the study is asked to break the main claim, with access to code, data and preregistration. Their report is answered in writing.
• Structured external review. Partners and domain experts review the model structure and the claims, using a form that asks for the strongest objection, not a general impression.
• Adversarial collaboration. Where a credible researcher holds an opposing view (for example, that topology-based prioritization is sufficient), design a joint test whose outcomes both sides agree in advance would change their view.
• Pre-mortem. Imagine the thesis was rejected at defense; write the examiner's reasons. Then test each reason.

All critique and responses go into the thesis's revision record. Examiners trust work whose weaknesses have visibly been attacked.

6.2 The red-team report on the Cascade Study (synthetic)

An internal red team produced five main objections:

1. Circularity. "DEP is computed from the model's own buffers and dependency map and then evaluated in the same model. Of course it does well. In reality the planner's map is wrong."
2. Short-buffer artefact. "Riverbend's buffers are short relative to restoration times. With realistic fuel contracts nothing you do matters."
3. Context selection. "Port Halvard was chosen to make the mechanism operate. Two regions, one picked for the method, cannot estimate how often DEP helps."
4. Heuristic weakness. "Your own oracle shows DEP is far from optimal in Riverbend. Why should anyone adopt a rule you know is suboptimal?"
5. Dual use. "You have shown that critical-service loss depends on which feeder is restored first and on whether the planner's map is right. That tells an adversary what to attack: the restoration-priority data."

Each objection is a hypothesis about the result. The rest of the chapter tests them.

6.3 Practical activity: boundary conditions, mismatch and many regions

The lab imports the Lab 5 simulator (save it as lab5_mc.py in the same folder).

# Lab 6 - boundary conditions, model mismatch and many synthetic regions (offline)
# Save Lab 5 as lab5_mc.py in the same folder first.
import random, statistics
import lab5_mc as m

def mean_gain(region, **kw):
    _, res, cust = m.experiment(region, n=300, oracle=False, **kw)
    return (statistics.fmean(res["DEP"]) - statistics.fmean(res["CRITFLAG"]),
            statistics.fmean(res["DEP"]) - statistics.fmean(res["SILO"]))

print("1) Riverbend buffer scaling: (DEP-CRITFLAG, DEP-SILO) mean loss")
for s in (0.5, 1.0, 2.0, 4.0):
    a, b = mean_gain(m.RIVERBEND, buffer_scale=s)
    print(f"   scale {s:3.1f}: {a:7.2f} {b:7.2f}")

print("2) Model mismatch: water plant truly on F1, planner believes F3")
true = {**m.RIVERBEND, "services": {**m.RIVERBEND["services"], "WATER": ("F1",) + m.RIVERBEND["services"]["WATER"][1:]}}
_, res, _ = m.experiment(true, n=300, oracle=False, believed_feeder={"WATER": "F3"})
print("   mean loss SILO %.2f CRITFLAG %.2f DEP %.2f" % tuple(statistics.fmean(res[k]) for k in ("SILO", "CRITFLAG", "DEP")))

print("3) 200 random synthetic regions: gain of DEP over CRITFLAG vs rank divergence")
def random_region(rng):
    feeders = [f"G{i}" for i in range(1, 6)]
    cust = {f: rng.choice([500, 1000, 2000, 4000, 8000]) for f in feeders}
    crit = rng.sample(feeders, 3)
    services = {}
    for i, f in enumerate(crit):
        lo = rng.choice([1, 4, 12, 36])
        services[f"S{i}"] = (f, (lo, lo * 3, lo * 2), rng.choice([3.0, 4.0, 5.0]), None)
    return {"customers": cust, "crews": 1, "dur": (2, 6, 4), "services": services}
def divergence(region):
    flag = [f for f in m.critflag(region) if f in {v[0] for v in region["services"].values()}]
    depo = [f for f in m.dep(region) if f in flag]
    return sum(1 for a, b in zip(flag, depo) if a != b)
rng = random.Random(2026)
rows = []
for _ in range(200):
    r = random_region(rng)
    _, res, _ = m.experiment(r, n=60, oracle=False)
    rows.append((divergence(r), statistics.fmean(res["DEP"]) - statistics.fmean(res["CRITFLAG"])))
for d in sorted({d for d, _ in rows}):
    g = [x for dd, x in rows if dd == d]
    print(f"   positions differing={d}: regions={len(g):3d} mean gain={statistics.fmean(g):7.2f} share DEP worse={sum(x > 0 for x in g)/len(g):.2f}")

Expected output:

1) Riverbend buffer scaling: (DEP-CRITFLAG, DEP-SILO) mean loss
   scale 0.5:    6.10  -86.14
   scale 1.0:    0.38  -47.57
   scale 2.0:    0.02   -7.25
   scale 4.0:    0.00    0.00
2) Model mismatch: water plant truly on F1, planner believes F3
   mean loss SILO 29.41 CRITFLAG 26.21 DEP 22.29
3) 200 random synthetic regions: gain of DEP over CRITFLAG vs rank divergence
   positions differing=0: regions= 69 mean gain=   0.00 share DEP worse=0.00
   positions differing=2: regions= 77 mean gain= -11.21 share DEP worse=0.01
   positions differing=3: regions= 54 mean gain= -23.97 share DEP worse=0.00

6.3.1 Objection 1: circularity

In part 2 the true water plant is on F1, but both the critical-customer list and the DEP planner believe it is on F3. Every criticality-aware policy degrades sharply: DEP from 4.32 to 22.29, CRITFLAG from 3.97 to 26.21. SILO actually improves (52.75 → 29.41) because it restores F1 first for customer reasons and thereby happens to restore water. The objection is sustained in a stronger form than the red team stated: the value of any criticality-aware ordering depends on the accuracy of the dependency map. The revision is to make map accuracy a stated precondition of the claim, to add a map-audit procedure to the method and to recommend measuring map error in any field application. This turns a weakness into a contribution: a measurable prerequisite that utilities can check.

6.3.2 Objection 2: short buffers

Part 1 scales every buffer. As buffers double, DEP's advantage over SILO falls from 47.57 to 7.25 weighted service-hours and disappears at four times the elicited buffers. The objection is sustained as a boundary condition: when buffers comfortably exceed total restoration time, ordering does not matter for critical services. This is exactly the mechanism's prediction in Chapter 3, so the result supports the mechanism while limiting the claim's scope. It also reveals something the red team did not raise: at half the elicited buffers, DEP is worse than CRITFLAG by 6.10 — the heuristic weakness of objection 4 grows when buffers are short, because mistakes about which feeder protects more weighted services cost more.

6.3.3 Objection 3: context selection

Part 3 generates 200 synthetic regions with five feeders, three critical services, random customer counts, buffer ranges and weights, and classifies them by how many positions differ between the CRITFLAG and DEP orders among critical feeders. Where the orders coincide (69 regions), the gain is exactly zero — as it must be. Where they differ in two positions (77 regions), DEP reduces mean loss by 11.21 and is worse in about 1% of regions; where they differ in three (54 regions), by 23.97, never worse in this sample. The pattern predicted by H3 holds across contexts that were not chosen for the method.

The red team's objection is nevertheless partly sustained: the generator draws customer counts and buffers independently, and that assumption fixes how often divergence occurs. Real regions may have correlations in either direction. If large feeders tend to host facilities with long buffers (large hospitals with large fuel stores), CRITFLAG's largest-first order runs opposite to DEP's shortest-buffer-first order and divergence becomes more common; if large feeders tend to host short-buffer services (dense-area telecom hubs on battery), the two orders agree and divergence becomes rarer. The frequency of divergence in real regions is an empirical question the thesis cannot answer with synthetic generators; the claim must be conditional on divergence, which a utility can measure about itself.

6.3.4 Objection 4: a known-suboptimal heuristic

Sustained. The oracle bound shows that a rule using weights and fan-out — for example, ordering feeders by the weighted value of the services they protect per hour of restoration, accounting for dependencies — could do better. The candidate may develop such a rule, but it must be labelled exploratory: it would be designed after seeing the confirmatory results and must be confirmed on new configurations (new regions, new seeds, preregistered) before any claim is made. The thesis can legitimately present it as the most promising next step.

6.4 Sensitivity analysis beyond one factor at a time

Varying one factor at a time (buffers, then weights, then durations) misses interactions — the buffer-scaling result shows that the heuristic's weakness interacts with buffer length. Global sensitivity analysis varies all uncertain inputs together and apportions output variance to inputs and their interactions; Saltelli et al. (2008) is a standard primer. For a thesis, a practical sequence is: one-at-a-time screening to find inputs that matter; a structured design (for example, a factorial over the few that matter); then, if resources allow, variance-based indices. Report what the conclusion depends on, in the reader's terms: "DEP's advantage over critical-flag ordering requires divergence between buffer and customer orders, a dependency map without feeder errors, and buffers shorter than about twice the expected restoration time."

Value sensitivity is distinct from parameter sensitivity. The predeclared weights encode one panel's priorities. Re-run with alternative weight sets — equal weights, weights from a second panel, weights that include ordinary customers — and report whether conclusions change (EX034).

6.5 Negative results and narrowing

The Cascade Study now has a narrowed claim:

In synthetic distribution-outage scenarios requiring sequential manual restoration, ordering by critical-service time-to-impact reduces predeclared weighted critical-service loss relative to critical-customer-flag ordering only when the two orders differ, the dependency map is accurate and buffers are short relative to restoration time; where these conditions hold, the reduction is large and the additional customer-hours relative to flag ordering are within a 15% margin. Any criticality-aware ordering protects critical services far better than customer-first ordering, at a customer-hour cost of roughly 60–70% in the regions studied, which exceeds the predeclared margin and is a policy trade-off rather than a technical improvement. The earliest-time-to-impact heuristic can underperform flag ordering when buffers are very short, because it ignores weights and fan-out.

Notice what the narrowed claim does: it states conditions a utility can measure; it reports the failed margin; it reports where the method loses; and it separates the robust finding (criticality-aware ordering versus customer-first) from the conditional contribution (timing versus lists). A negative result about H2 in Riverbend is not hidden; it is the evidence that defines the boundary.

Publish negative results usefully. State the hypothesis, the design, the effect size with its interval and the smallest effect of interest it rules out. "No significant difference" is useless; "a reduction of 10% or more relative to flag ordering is ruled out in Riverbend" is useful.

6.6 Dual-use review of the findings

Applying the three questions of Chapter 1 (§1.4.3):

Lesson table; scroll horizontally to read all columns →

The second row illustrates a recurring pattern in infrastructure research: a resilience finding often doubles as an attack-surface finding. The relevant threat context is public — CISA's AA24-038A describes pre-positioning in infrastructure IT networks, where planning data typically lives — so the defensive recommendation (integrity protection and audit of restoration-priority data) has clear value. Coordinate with partners and, where appropriate, the relevant sector information-sharing body before publication (Chapter 8).

6.7 Responding to reviewers

A good response letter quotes each comment, states whether it is accepted, partly accepted or not accepted, describes the change (with location), and — when not accepted — gives evidence rather than opinion. Most importantly, it changes claims, not just wording. Compare:

• Defensive: "We thank the reviewer. Port Halvard was selected on predeclared criteria, so selection bias is not a concern."
• Responsive: "Partly accepted. Port Halvard was selected on predeclared mechanism-derived criteria, which makes it a fair test of the mechanism but not an estimate of how often DEP helps. We added a 200-region analysis (§6.3.3), narrowed the claim to regions with order divergence and stated that the frequency of divergence in real regions is unknown (Limitations, item 4)."

6.8 Common errors and limits

• Treating peer review as a hurdle instead of a method.
• Evaluating a policy only in the model used to derive it.
• Reporting only one-factor-at-a-time sensitivity.
• Treating a random-context generator as a sample of reality.
• Presenting an exploratory revision as confirmatory.
• Writing negative results as "not significant".

• Skipping dual-use review because results are "only simulations".

• Accepting a red-team objection in its original form without testing whether a stronger version holds, as happened with circularity.

• Treating a boundary condition as a defeat rather than as a confirmed prediction of the mechanism that narrows the claim.
• Assuming a random-region generator's independence assumptions match real regions, where customer counts and buffers may be correlated.
• Writing a response letter that thanks reviewers and defends the original claim instead of changing it where the evidence requires.
• Publishing partner-derived buffer ranges because they look harmless in aggregate.

Exercises — chapter 6

EX033 — Red-team report. Objective: attack your own claim. Inputs: the Cascade Study claim before narrowing (Chapter 5 summary) or your own. Constraints: at least five objections, each stated as a testable hypothesis. Steps: (1) write each objection; (2) say what evidence would sustain or dismiss it; (3) design the test; (4) predict the outcome before running anything. Expected evidence: report. Rubric: 3 = objections specific and testable, including circularity and context selection, with predictions recorded before testing; 2 = one or two untestable; 1 = generic; 0 = none.

EX034 — Weight sensitivity. Objective: test value sensitivity. Inputs: Lab 5 and Lab 6. Constraints: offline. Steps: (1) set all service weights to 1.0 in both regions; (2) re-run the main comparison; (3) record DEP, CRITFLAG and SILO losses; (4) state whether each verdict of the hypothesis table changes; (5) explain why. Expected evidence: modified code, outputs, paragraph. Rubric: 3 = exact outputs, correct verdict analysis, explanation tied to fan-out and weights; 2 = outputs right, explanation thin; 1 = errors; 0 = not run.

EX035 — Map-error experiment. Objective: quantify fragility to map error. Inputs: Lab 6 part 2. Constraints: offline. Steps: (1) repeat the mismatch with the telecom office truly on F4 while the planners believe F2; (2) record the losses; (3) compare with part 2; (4) propose a map-audit procedure with a measurable accuracy target. Expected evidence: outputs and procedure. Rubric: 3 = correct outputs, comparison with the water mismatch, audit procedure measurable (sampled field verification, accuracy rate, owner sign-off); 2 = procedure vague; 1 = not run correctly; 0 = none.

EX036 — Generator critique. Objective: recognize the generator as a model. Inputs: random_region in Lab 6. Constraints: propose a change that encodes a plausible real-world correlation. Steps: (1) modify the generator so that larger feeders tend to host critical services with longer buffers; (2) re-run part 3; (3) compare the distribution of divergence and gains with the original part 3; (4) predict, then check, what the opposite correlation would do; (5) explain what this means for the claim. Expected evidence: code, output, paragraph. Rubric: 3 = correlation implemented sensibly; learner notices that this correlation makes CRITFLAG and DEP orders run in opposite directions, so divergence becomes more frequent (regions with no divergence fall sharply), and explains that the opposite correlation would make it rarer, so the claim must stay conditional on measured divergence; 2 = implementation sound but the direction of the effect misread; 1 = no comparison; 0 = none.

EX037 — Dual-use decision. Objective: decide publication treatment. Inputs: four findings from your own research or: (a) a list of water utilities whose SCADA is reachable via a specific vendor's remote-access product; (b) the observation that remote-access products are a common entry point across sectors; (c) a method to estimate time-to-impact from public facility data; (d) the time-to-impact estimates for named hospitals in one city. Constraints: the three questions of §1.4.3. Steps: (1) assess uplift for each finding; (2) assess defensive value; (3) decide among publish, generalize, delay and restrict; (4) state who should be notified before publication. Expected evidence: table. Rubric: 3 = (a) restrict and notify via coordinated disclosure, (b) publish, (c) publish with care, (d) withhold or restrict, with sound reasoning; 2 = one misjudged; 1 = two or more; 0 = none.

EX038 — Response letter. Objective: change claims in response to critique. Inputs: the five red-team objections in §6.2. Constraints: one response per objection; at least two must lead to a changed claim. Steps: (1) quote each objection; (2) mark it accepted, partly accepted or not accepted; (3) describe the change and its location; (4) give evidence for any non-acceptance. Expected evidence: response letter. Rubric: 3 = responsive rather than defensive, claims changed where warranted, exploratory work labelled; 2 = mostly responsive; 1 = defensive; 0 = none.

Chapter synthesis

Critique is part of the method. The red team's objections became hypotheses and tests. Circularity was sustained in a stronger form — every criticality-aware policy depends on an accurate dependency map — which became a measurable precondition and a defensive recommendation. Short buffers were confirmed as a boundary condition predicted by the mechanism, and very short buffers exposed the heuristic's weakness. A 200-region analysis supported the mechanism across contexts not chosen for it while reminding us that a generator is a model. The narrowed claim states measurable conditions, reports the failed margin and separates the robust finding from the conditional contribution. Dual-use review showed that a resilience finding can also be an attack-surface finding and decided what to publish at what level.