Chapter 6 — Web3, blockchain and cryptoasset security: controlled evaluation
Lesson map
Lesson 06 published. Map, example, EX026.
flowchart TD
N0["Learning outcomes"]
N1["6.1 Designing the comparison"]
N2["Lab — Sign-off design comparison"]
N3["6.2 Interpreting results"]
N4["6.3 Cost and detection together"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX026"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["6.1 Designing the comparison"]
N2["Lab — Sign-off design comparison"]
N3["6.2 Interpreting results"]
N4["6.3 Cost and detection together"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX026"]
[Learning outcomes]
|
v
[6.1 Designing the comparison]
|
v
[Lab — Sign-off design comparison]
|
v
[6.2 Interpreting results]
|
v
[6.3 Cost and detection together]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX026]
Learning outcomes
You will (1) design a controlled comparison; (2) run it on seeded synthetic data; (3) interpret detection and cost together.
6.1 Designing the comparison
Nadia evaluates how settlement batches should be signed off before the multisig signs them. Design A uses one reviewer checking the batch file. Design B uses two reviewers from different institutions and an automatically generated summary showing totals, recipients and whether any transfer repeats an earlier batch. She builds eight synthetic batches, four containing seeded errors of different kinds, and records whether each design catches each error and how long sign-off takes. A controlled comparison on the same batches isolates the effect of the design itself rather than the particular batches each design happened to see.
Lab — Sign-off design comparison
"""Compare two settlement sign-off designs on seeded synthetic batches (read-only)."""
# batch id, seeded error (None or kind), minutes to sign under A (one reviewer), B (two reviewers + summary)
BATCHES = [
("S01", None, 6, 11), ("S02", "amount", 5, 12), ("S03", None, 7, 10),
("S04", "duplicate", 6, 13), ("S05", None, 5, 9), ("S06", "recipient", 8, 14),
("S07", None, 6, 10), ("S08", "duplicate", 7, 12),
]
CAUGHT_A = {"amount"} # single reviewer catches only amount errors
CAUGHT_B = {"amount", "duplicate", "recipient"}
def rate(caught):
seeded = [b for b in BATCHES if b[1]]
return sum(1 for b in seeded if b[1] in caught), len(seeded)
def main():
for name, caught, col in (("A", CAUGHT_A, 2), ("B", CAUGHT_B, 3)):
hit, n = rate(caught)
mins = sorted(b[col] for b in BATCHES)
median = (mins[3] + mins[4]) / 2
print(f"design {name}: caught {hit}/{n}; median sign-off {median} min")
if __name__ == "__main__":
main()
Run with python3 lab_signoff.py. Verified output:
design A: caught 1/4; median sign-off 6.0 min
design B: caught 4/4; median sign-off 11.5 min
6.2 Interpreting results
The lab shows design A catching 1 of 4 seeded errors with a median sign-off time of 6 minutes, and design B catching all 4 with a median of 11.5 minutes. Design B nearly doubles the time but catches the duplicate and recipient errors that a lone reviewer missed. Nadia notes that the results come from eight synthetic batches and seeded errors chosen by her, so they show the mechanism rather than precise real-world rates. She recommends design B and a later measurement on real pilot batches to confirm that time and detection behave similarly.
6.3 Cost and detection together
Reporting only detection would favour design B without question; reporting only time would favour A. Nadia presents both and estimates the cost: with about 40 batches a week, design B adds roughly three hours of reviewer time weekly across the institutions. The committee judges this worthwhile given settlement values. She also notes that the summary, not just the second reviewer, likely drives detection of duplicates, which suggests a cheaper option to test later: one reviewer with the summary. Interpreting results together with cost keeps evaluation useful for real decisions.
Worked example
Design B catches 4 of 4 seeded errors at a median 11.5 minutes against A's 1 of 4 at 6 minutes, adding about three reviewer hours a week.
Synthetic case. The batches, errors and times are synthetic.
Errors and limits
Presenting detection without cost, or cost without detection, biases the decision.
Exercises
- EX026 — Design B description. What does design B add? Rubric: 3 = two reviewers from different institutions and a summary;; 2 = two reviewers;; 1 = vague;; 0 = none.
- EX027 — Detection results. How many errors do A and B catch? Rubric: 3 = A 1 of 4, B 4 of 4;; 2 = 1/4 and 4/4;; 1 = both 4;; 0 = none.
- EX028 — Median times. What are the median sign-off times? Rubric: 3 = 6 and 11.5 minutes;; 2 = 6/11.5;; 1 = equal;; 0 = none.
- EX029 — Weekly cost. What weekly cost does design B add? Rubric: 3 = about three hours of reviewer time;; 2 = three hours;; 1 = vague;; 0 = none.
- EX030 — Cheaper variant. What cheaper option is suggested for later testing? Rubric: 3 = one reviewer with the summary;; 2 = summary only;; 1 = no review;; 0 = none.
Synthesis
Controlled comparison on seeded data shows mechanisms, and cost-aware interpretation guides choice.
Continue studying — chapter 06 of 12
← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →