Chapter 23 — Web3, blockchain and cryptoasset security: evaluation
Lesson map
Lesson 23 published. Map, example, EX067.
flowchart TD
N0["Learning outcomes"]
N1["23.1 Setting evaluation goals"]
N2["23.2 Seeded tests"]
N3["23.3 Reporting strengths and gaps"]
N4["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4
N4 --> EX["Worked example"]
EX --> Q["Exercise EX067"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["23.1 Setting evaluation goals"]
N2["23.2 Seeded tests"]
N3["23.3 Reporting strengths and gaps"]
N4["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX067"]
[Learning outcomes]
|
v
[23.1 Setting evaluation goals]
|
v
[23.2 Seeded tests]
|
v
[23.3 Reporting strengths and gaps]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX067]
Learning outcomes
You will (1) evaluate controls and monitoring against goals; (2) use seeded tests; (3) report strengths and gaps.
23.1 Setting evaluation goals
After a year Lucas evaluates the makerspace's ledger controls and monitoring. Goals were agreed earlier: every credit grant matched to a workshop, every privileged event matched to an approval, node lag detected within five minutes, and restores within one hour. He collects evidence from logs, drills and reports for each goal. Evaluating against pre-agreed goals keeps the evaluation honest, because it cannot be adjusted to fit results after the fact, and it gives the board a clear picture of whether the investment in security is paying off.
23.2 Seeded tests
To test detection, Lucas seeds synthetic problems into the test environment: an unmatched grant, a privileged event without a ticket, a node stopped to create lag and a configuration change. The monitoring detects all four, with node lag found in three minutes. He also seeds a subtle case: a grant matched to a workshop recorded for the wrong member. Monitoring misses it, because matching checks only workshop identifiers. Seeded tests show what monitoring really catches, including gaps that normal operation would never reveal because the problem had simply not happened yet.
23.3 Reporting strengths and gaps
Lucas reports that all four main goals are met and describes the gap: grants matched to the wrong member pass unnoticed. He recommends extending matching to check member identity as well as workshop, and adding a monthly sample review by the coordinator. He presents strengths and gaps together with evidence. The board approves the recommendations. Ms Adebayo notes that evaluations which only confirm success are of limited use; finding a gap is the most valuable outcome, because it shows exactly where to improve next.
Worked example
Monitoring detects all four seeded problems, node lag within three minutes, but misses a grant matched to the wrong member, leading to identity-aware matching.
Synthetic case. The evaluation and seeded tests are synthetic.
Errors and limits
Evaluating only with problems monitoring was designed for hides the gaps it cannot see.
Exercises
- EX067 — Lag detection time. How quickly was node lag detected? Rubric: 3 = three minutes;; 2 = 3 minutes;; 1 = an hour;; 0 = none.
- EX068 — Missed case. Which seeded case was missed? Rubric: 3 = a grant matched to the wrong member;; 2 = wrong member;; 1 = unmatched grant;; 0 = none.
- EX069 — Gap fix. What is recommended to close the gap? Rubric: 3 = match member identity as well as workshop;; 2 = identity matching;; 1 = vague;; 0 = none.
Synthesis
Goal-based evaluation with seeded tests reveals real strengths and gaps.
Continue studying — chapter 23 of 30
← Chapter 22 (draft) · Course chapter index · Chapter 24 (draft) →