Chapter 4 — Baseline replication and pilot
Lesson map
Lesson 04 published. Map, example, EX021.
flowchart TD
N0["Learning outcomes"]
N1["4.1 Replicating the technique"]
N2["Lab — Monte Carlo annual loss for ransomware on reco"]
N3["4.2 Reading the output"]
N4["4.3 Piloting calibration"]
N5["4.4 Adjusting the protocol"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX021"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["4.1 Replicating the technique"]
N2["Lab — Monte Carlo annual loss for ransomware on reco"]
N3["4.2 Reading the output"]
N4["4.3 Piloting calibration"]
N5["4.4 Adjusting the protocol"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX021"]
[Learning outcomes]
|
v
[4.1 Replicating the technique]
|
v
[Lab — Monte Carlo annual loss for ransomware on reco]
|
v
[4.2 Reading the output]
|
v
[4.3 Piloting calibration]
|
v
[4.4 Adjusting the protocol]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX021]
Learning outcomes
You will (1) replicate a published quantitative technique on synthetic data; (2) pilot calibration training with assessors; (3) adjust the protocol from pilot evidence.
4.1 Replicating the technique
Amara first replicates the core FAIR-style technique on synthetic data so she understands it fully. For the ransomware scenario she defines ranges for loss event frequency and loss magnitude, using minimum, most likely and maximum values, and runs a Monte Carlo simulation of twenty thousand years. The lab reports the mean annual loss, the chance of any loss in a year and high percentiles. She checks that the results behave as expected when ranges change, which is the basic test that a simulation is implemented correctly.
Lab — Monte Carlo annual loss for ransomware on records
# FAIR-style Monte Carlo: annual loss for ransomware on records (synthetic ranges).
import random
r = random.Random(5); N = 20000
def tri(lo, mode, hi): return r.triangular(lo, hi, mode)
losses = []
for _ in range(N):
freq = tri(0.02, 0.06, 0.20) # loss events per year
events = sum(1 for _ in range(3) if r.random() < freq)
losses.append(sum(tri(300, 1500, 6000) for _ in range(events)))
losses.sort()
mean = sum(losses) / N
print(f"mean annual loss {mean:.0f}k; P(loss>0) {sum(l > 0 for l in losses) / N:.3f}; 95th pct {losses[int(.95 * N)]:.0f}k; 99th pct {losses[int(.99 * N)]:.0f}k")
print(f"P(loss > 2000k) {sum(l > 2000 for l in losses) / N:.3f}")
Run with python3 lab_mc.py. Verified output:
mean annual loss 709k; P(loss>0) 0.247; 95th pct 4062k; 99th pct 5810k
P(loss > 2000k) 0.163
4.2 Reading the output
The simulation gives a mean annual loss of about 709 thousand, but in roughly three years out of four there is no loss at all, and the 95th percentile is about 4.06 million. The chance of losing more than two million in a year is about 16%. Amara notes that the mean hides this skew: a board told only the average would misunderstand the risk. She plans to present loss exceedance probabilities, which answer questions boards actually ask, such as how likely a loss above a reserve level is.
4.3 Piloting calibration
Amara pilots a two-hour calibration module with six volunteer assessors. They practise giving ninety percent ranges for general-knowledge questions and receive feedback on how often the true answer fell inside. Before training, their ranges contained the answer about half the time; afterwards, closer to eight in ten. Hubbard and Seiersen (2016) describe this kind of training. Assessors report that the exercise was uncomfortable but useful, and two say it changed how they think about uncertainty in their own risk estimates. All six complete it.
4.4 Adjusting the protocol
The pilot leads to three changes. The calibration module is shortened to ninety minutes because attention dropped in the second hour. Scenario descriptions are rewritten to remove clinical jargon that confused non-clinical assessors. The analysis plan adds loss exceedance at a fixed threshold as a secondary outcome, because pilot assessors found it more meaningful than the mean. Amara records each change and its reason in an amendment log submitted to the ethics board, keeping the pre-registered primary outcomes unchanged. The board approves.
Worked example
Amara replicates a Monte Carlo loss model, reads its skewed output, pilots calibration training and records protocol amendments.
Synthetic case. Ranges, results and pilot figures are synthetic; Hubbard and Seiersen (2016) is real.
Errors and limits
Simulations depend on input ranges. Pilot gains may not persist.
Exercises
- EX021 — Simulation size. How many simulated years did the lab run? Rubric: 3 = twenty thousand;; 2 = 20,000;; 1 = vague;; 0 = none.
- EX022 — Mean annual loss. What mean annual loss did the lab report? Rubric: 3 = about 709 thousand;; 2 = 709k;; 1 = vague;; 0 = none.
- EX023 — No-loss years. In roughly what share of years was there no loss? Rubric: 3 = about three in four;; 2 = 75%;; 1 = vague;; 0 = none.
- EX024 — Exceedance figure. What was the chance of losing more than two million? Rubric: 3 = about 16%;; 2 = 0.163;; 1 = vague;; 0 = none.
- EX025 — Pilot calibration. How did range hit rates change with training? Rubric: 3 = from about half to about eight in ten;; 2 = improved;; 1 = vague;; 0 = none.
- EX026 — Unchanged outcomes. What did Amara keep unchanged when amending the protocol? Rubric: 3 = the pre-registered primary outcomes;; 2 = primary outcomes;; 1 = vague;; 0 = none.
Synthesis
Replication, piloting and logged amendments prepare a sound main study.
Continue studying — chapter 04 of 08
← Chapter 03 (draft) · Course chapter index · Chapter 05 (draft) →