OpenCyber Academy
Postgraduate · Chapter 6 · English
Course code: OCA-157Chapter 06: OCA-157-C006

Chapter 6 — Controlled evaluation

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 06 published. Map, example, EX026.

flowchart TD
  N0["Learning outcomes"]
  N1["6.1 Evaluation targets"]
  N2["6.2 Scoring lab"]
  N3["6.3 Pre-registration"]
  N4["6.4 GAP language"]
  N5["6.5 Independence"]
  N6["6.6 Ethics"]
  N7["6.7 Narrative"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX026"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["6.1 Evaluation targets"]
  N2["6.2 Scoring lab"]
  N3["6.3 Pre-registration"]
  N4["6.4 GAP language"]
  N5["6.5 Independence"]
  N6["6.6 Ethics"]
  N7["6.7 Narrative"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX026"]

[Learning outcomes]
    |
    v
[6.1 Evaluation targets]
    |
    v
[6.2 Scoring lab]
    |
    v
[6.3 Pre-registration]
    |
    v
[6.4 GAP language]
    |
    v
[6.5 Independence]
    |
    v
[6.6 Ethics]
    |
    v
[6.7 Narrative]
    |
    v
[Worked example] --> [Exercise EX026]

Learning outcomes

You will (1) design evaluations of scenarios and controls; (2) use scoring that balances learning and risk; (3) pre-register; (4) report gaps without doom.

6.1 Evaluation targets

Scenario choice, control coverage, detection rules and programme outcomes each need different evaluation designs. Amira refuses a single score that pretends to measure them together. The lab scenario ranking is for scenario choice; detection rules need purple-team measures; programme outcomes need decision records such as DEC-14. Mixing scenario ranks, purple-team measures and DEC-14 outcomes into one metric would give the board a number that measures none of them, which is how board packs lose meaning. Each evaluation target therefore keeps its own design and its own measure in her pack.

6.2 Scoring lab

lab_scenario ranks synthetic scenarios with score = 2×learning − risk so high-learning, manageable-risk options surface.

"""Lab 6: score synthetic TIBER-style scenario options by detection learning vs ops risk (1-3)."""
opts = [("A","Stolen VPN + SWIFT-adjacent pivot",3,3),("B","Public web defacement only",1,1),
        ("C","Supply-chain library in batch app",3,2),("D","Branch Wi-Fi rogue AP",2,2)]
for i,t,learn,risk in sorted(opts, key=lambda x:(-(x[2]*2-x[3]), x[0])):
    print(f"{i} score={learn*2-risk:+d} learn={learn} risk={risk} {t}")

Run with python3 lab_scenario.py. Verified output:

C score=+4 learn=3 risk=2 Supply-chain library in batch app
A score=+3 learn=3 risk=3 Stolen VPN + SWIFT-adjacent pivot
D score=+2 learn=2 risk=2 Branch Wi-Fi rogue AP
B score=+1 learn=1 risk=1 Public web defacement only

The supply-chain library in the batch app (C) scores highest at +4; the stolen VPN with SWIFT-adjacent pivot (A) scores +3 but carries risk 3; the branch Wi-Fi rogue AP (D) scores +2; and public web defacement (B) scores only +1. The rank is an input, not a funding decision. Amira demotes a high lab score when white-team feasibility is missing, and feasibility still decides funding after the lab rank.

6.3 Pre-registration

Metrics and thresholds are tagged in git before results, so Amira cannot move the goalposts after seeing scores. The pre-registration states the primary measure, the threshold and the exclusions, and it is short on purpose: a peer should be able to find the tag and read it without asking her. Changing a threshold after seeing scores requires an amended tag and a decision-log note that names why. Pre-registration is how Meridian Clearing House keeps evaluation honest when favourite scenarios underperform and the temptation to redefine success is strongest.

6.4 GAP language

"T1190 0/2 GAP" schedules engineering; it does not claim permanent blindness. T1190 0/2 means neither of the two exploit-public-facing-application attempts was detected, a sample far too small to estimate a rate. A dated engineering ticket is the honest companion to a GAP label. Saying "we are blind forever" is neither true nor useful when a dated ticket can close the gap, and doom language trains directors to ignore gaps that engineering could close. Amira's language in findings and board packs therefore stays in the lane where a GAP schedules engineering work.

6.5 Independence

The white team or a peer runs the scoring on the same inputs. In the scenario workshop the peer receives the four options with their learning and risk ratings, but not Amira's ranking. Independence is not optional politeness: it is how optimistic grading of favourite scenarios is caught before funding decisions lock in. Disagreements are not averaged away; each keeps both ratings and a one-line reason from each scorer, and becomes a design note tagged FEASIBILITY for the next workshop. Amira would rather lose a preferred scenario than keep a score only she can defend.

6.6 Ethics

Scenarios that need live participant impact are rejected at design time. Live harm to participants is not an acceptable price for a more realistic exercise at Meridian Clearing House, and synthetic clearing templates and isolated ranges are enough for learning. Scenario A's SWIFT-adjacent pivot is therefore played only against an isolated range, never against systems carrying live clearing traffic. The ethics cover sheet says synthetic-only on page one of every project, which makes live participant impact a design-time rejection rather than a negotiation in the middle of an exercise.

6.7 Narrative

Feasibility is part of the score, not an afterthought. White-team feasibility for scenario C includes a library inventory owner who can halt deploys of the batch application inside the exercise window, not merely approve a ticket later. Without that owner, Amira demotes C despite its +4 lab score. Independent scoring means she leaves the room while a peer applies lab_scenario to the same inputs. When a white-team view overrides a lab score, as with scenario C, she records the reason as a design note filed beside the lab output rather than changing the score silently.

6.8 Feasibility notes

Design notes from scoring disagreements are tagged FEASIBILITY so they appear in the next scenario workshop automatically, turning a scoring dispute into an input for designing the next scenario. Each note records the scenario letter, both scores, the missing feasibility element and who must supply it before the workshop. Scenario C's missing library owner was filed this way and became the opening item of the following workshop. Amira reads the tagged notes aloud, so the peer who disagreed sees their point land and juniors learn that losing a favourite option is normal when stop owners are missing.

6.9 Board slide

Amira communicated the feasibility demotion of scenario C to the board as strength, not indecision. Her slide title was "We refused a high-scoring scenario we could not stop", and it paired C's +4 lab score with the missing deploy-freeze owner, so the reason fitted on one line. The board applauded. The refusal was recorded in the decision log beside the lab score, so the reason survives the meeting. Lab scores without stop owners are now labelled provisional in every packet that leaves her desk.

6.10 Provisional labels

Provisional labels on scores are bright orange in the PDF export, so a provisional rank cannot be quoted as final once the PDF leaves the team. Only scores with white-team feasibility confirmed may lose the orange mark, and confirmation means the white-team lead has signed the feasibility line, including a named stop owner for that scenario. Directors have thanked Amira for the orange colour when a vendor tried to cite a provisional rank as a commitment. The label costs nothing to apply and prevents a ranking from being treated as an approval.

Worked example (synthetic MCH)

lab_scenario ranks supply-chain library scenario highest among the four; Amira still subjects it to white-team feasibility before funding.

Synthetic case — Meridian Clearing House (MCH). The clearing organisation, Amira Hassan and all figures are invented for teaching.

Errors and limits

Threat-led and clearing-system tests require specialised legal authorization. This book uses synthetic scenarios only and never supplies exploit code or live targeting advice for market infrastructures.

Exercises

Synthesis

Controlled evaluation ranks learning against risk and refuses doom language for small samples.

Continue studying — chapter 06 of 12

← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →

Worked answers

EX026. C — Supply-chain library in batch app.

EX027. score = 2×learning − risk.

EX028. Ranking informs; white-team feasibility and broader criteria still decide funding.

EX029. As a gap that schedules engineering work, not as permanent blindness.

EX030. In a dated git tag or equivalent registration before results.