OpenCyber Academy
Master's · Chapter 8 · English
Course code: OCA-180Chapter 08: OCA-180-C008

Chapter 8 — Independent review and defense

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 08 published. Map, example, EX045.

flowchart TD
  N0["Learning outcomes"]
  N1["8.1 Independent review"]
  N2["8.2 Defence posture"]
  N3["8.3 Dual-use challenges"]
  N4["8.4 Dissemination"]
  N5["8.5 Closing the placement"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX045"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["8.1 Independent review"]
  N2["8.2 Defence posture"]
  N3["8.3 Dual-use challenges"]
  N4["8.4 Dissemination"]
  N5["8.5 Closing the placement"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX045"]

[Learning outcomes]
    |
    v
[8.1 Independent review]
    |
    v
[8.2 Defence posture]
    |
    v
[8.3 Dual-use challenges]
    |
    v
[8.4 Dissemination]
    |
    v
[8.5 Closing the placement]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX045]

Learning outcomes

You will (1) prepare materials for independent review; (2) answer methods and ethics questions under defence; (3) handle dual-use challenges; (4) plan revisions and dissemination; and (5) close the placement cleanly.

8.1 Independent review

Share the dissertation, synthetic appendix and ethics package with an internal reviewer who did not supervise weekly. Ask them to check: measure–claim fit; blinding; tag audit; whether any section drifts into attack instruction.

Choosing a reviewer who did not supervise weekly matters because a weekly supervisor has already accepted Noah's framing. A fresh reader is more likely to notice, for example, that the tag audit reached 96% agreement only after second-person review, and to ask how many tags were wrong before that check. The reviewer's written comments go into the response trail alongside Noah's replies.

8.2 Defence posture

Open with question, design, primary result and conditions. Expect questions on randomisation, severity balance, why synthetic segments, and what would falsify the result. Bring a one-page methods cheat sheet.

For Noah, the opening fits in about ninety seconds: does ATT&CK tagging raise 30-day remediation on CBM's synthetic segments; stratified randomisation by severity and asset class; 62% against 44%, a difference of about 18 points with an interval excluding zero; synthetic training segments, volunteer remediators and one organisation. The cheat sheet adds what would falsify the result, such as a tag audit showing that correct tags made no difference.

8.3 Dual-use challenges

If asked to "show how the finding was exploited", refuse and show the report template fields instead. Reaffirm authorization boundaries and coordinated disclosure for any real issues encountered during placement.

A demand for exploit detail at a defence is usually a test of judgement, not a request for data. Noah's answer has four parts: he declines; he shows the template, which deliberately has no field for exploit steps; he explains that his authorization covered synthetic training assessments only; and he notes that any real issue found during the placement would go through CBM's coordinated disclosure route, not into a dissertation.

8.4 Dissemination

Submit a short paper to a measurement or SOUPS-like human-factors venue if appropriate; present the practitioner brief internally; release the synthetic appendix with hashes. Do not release client-identifying materials.

Each output has its own exclusions. The short paper reports the 30-day comparison and its limits but no CBM identifiers. The internal brief may name CBM teams because it stays inside CBM. The public appendix contains only the synthetic export, scripts and templates, released with hashes so later readers can confirm they hold the same files the examiners checked.

"""ATT&CK-style coverage matrix over synthetic authorized techniques (names only)."""
# Synthetic engagement log: technique_id -> observed (True/False)
PLAN = ["T1046", "T1595", "T1082", "T1083", "T1059", "T1078", "T1021"]
OBSERVED = {"T1046", "T1595", "T1082", "T1083", "T1078"}
LABELS = {
    "T1046": "Network Service Discovery",
    "T1595": "Active Scanning",
    "T1082": "System Information Discovery",
    "T1083": "File and Directory Discovery",
    "T1059": "Command and Scripting Interpreter",
    "T1078": "Valid Accounts",
    "T1021": "Remote Services",
}
covered = 0
for tid in PLAN:
    hit = tid in OBSERVED
    covered += hit
    print(f"{tid} {LABELS[tid]:34} {'COVERED' if hit else 'PLANNED-NOT-SEEN'}")
print(f"coverage={covered}/{len(PLAN)} ({100*covered/len(PLAN):.0f}%)")

Expected output:

T1046 Network Service Discovery          COVERED
T1595 Active Scanning                    COVERED
T1082 System Information Discovery       COVERED
T1083 File and Directory Discovery       COVERED
T1059 Command and Scripting Interpreter  PLANNED-NOT-SEEN
T1078 Valid Accounts                     COVERED
T1021 Remote Services                    PLANNED-NOT-SEEN
coverage=5/7 (71%)

8.5 Closing the placement

Return access; confirm synthetic data only remains in the archive; thank remediators; leave the ticket templates behind if CBM wants them. Coverage language from the lab becomes a teaching reminder: planned-not-seen items wait for the next authorized plan.

Closing cleanly is part of the evidence that the work was governed. Noah returns his assessment accounts on the last day, asks CBM to confirm their removal in writing, and checks that the archive holds only synthetic tickets. Offering the tagged templates is optional for CBM; adopting them in live work would need its own authorization, as the worked example below notes.

8.6 Worked example (synthetic CBM case)

At defence, Noah is asked whether ATT&CK tagging merely marks findings that were already well written. He points to stratified randomisation and the tag audit, and accepts that residual confounding is a limitation. He refuses a request for exploit detail and shows that the template has no field for exploit steps. CBM adopts tagged templates for training assessments pending a live pilot under separate authorization.

The answer to the confounding question is honest: randomisation and the audit reduce it, but cannot remove it.

8.7 Errors and limits

Each error undoes earlier care. Overclaiming live enterprise impact goes beyond synthetic training segments and one organisation. Real vulnerability details in a public appendix turn research into an attack resource. Privileged access left active after the placement is a risk to CBM, which is why Noah returns his accounts and asks for written confirmation. Check the defence and closeout plan for all three.

Exercises

Synthesis

Independent review and defence test whether your claims, ethics and artefacts stand without you in the room — and whether you leave the host organisation safer and cleaner than you found it.

Continue studying — chapter 08 of 08

← Chapter 07 (draft) · Course chapter index · Worked results ↓ · Next course via chapter index →

Worked answers

EX045. Measure–claim fit? Blinding held? Tag audit adequate? Any attack-instruction drift?

EX046. "I compared ATT&CK-tagged vs narrative findings on synthetic segments; 30-day remediation was 62% vs 44% across ~110 findings per arm under authorization."

EX047. "I will not show an exploit. Here is the template, which has no field for exploit steps. Ethics forbid attack procedures. Real issues go through coordinated disclosure."

EX048. Short paper (no client IDs); internal brief (no public vuln details).

EX049. Add to next authorized backlog with permission.

EX050. Return access; confirm data scope; thank staff; leave templates; revoke tokens.