Chapter 10 — Metrics and reporting to the board
Lesson map
Lesson 10 published. Map, example, EX046.
flowchart TD
N0["Learning outcomes"]
N1["10.1 Choosing metrics"]
N2["10.2 Medians and trends"]
N3["10.3 Misleading metrics"]
N4["10.4 The board page"]
N5["10.5 Decisions"]
N6["10.6 Honesty"]
N7["Audience"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX046"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["10.1 Choosing metrics"]
N2["10.2 Medians and trends"]
N3["10.3 Misleading metrics"]
N4["10.4 The board page"]
N5["10.5 Decisions"]
N6["10.6 Honesty"]
N7["Audience"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX046"]
[Learning outcomes]
|
v
[10.1 Choosing metrics]
|
v
[10.2 Medians and trends]
|
v
[10.3 Misleading metrics]
|
v
[10.4 The board page]
|
v
[10.5 Decisions]
|
v
[10.6 Honesty]
|
v
[Audience]
|
v
[Worked example] --> [Exercise EX046]
Learning outcomes
You will (1) choose metrics that show SOC value; (2) present trends and risks to non-specialists; (3) avoid misleading metrics; (4) link reporting to decisions.
10.1 Choosing metrics
Priya picks: log coverage of critical assets, time to acknowledge critical alerts, rule precision, open high risks and exercise findings closed. Each metric relates to a first-year goal: coverage measures what the SOC can see, acknowledgement time measures response at any hour, precision measures how much time rules waste, and open risks and exercise findings show whether problems are being closed. Priya defines each one in writing, including how it is counted, so the figures are comparable each month. She agrees the definitions with Tom and the provider, so the acknowledgement times in the provider's reports and in Tom's records are counted the same way.
10.2 Medians and trends
Medians and month-to-month trends avoid being misled by one unusual case. A single slow response could distort an average, while the median shows typical performance. Priya still reports the worst case, such as the 26-minute provider test, so that unusual cases are not hidden. Trends over several months show whether the SOC is improving, which matters more to the board than a single month's figure. Each chart on the board page shows at least six months. Tom checks each chart against the provider's acknowledgement records and his own triage notes before the page goes to the board.
10.3 Misleading metrics
"Alerts handled" looks impressive but grows with noise; Priya leaves it out. A rising count of handled alerts could mean more attacks, or simply noisier rules, so it does not show whether Fenwick is safer. After tuning, the macro rule fired far less, which would make the count fall even though detection improved. Priya chooses metrics that change in the right direction when the SOC gets better, such as precision and acknowledgement time. She explains this choice on the board page, so nobody asks why the count is missing.
10.4 The board page
One page: three charts, two decisions needed, one sentence per risk. Keeping it to one page makes the board page readable in a meeting. The three charts show coverage, acknowledgement time and precision over time. The decisions needed tell the board what it is being asked to approve, such as funding for ERP log collection. One sentence per risk gives the board enough to understand each open risk without technical detail. Priya prepares the page from the same data as the monthly report, and Tom checks the figures before it goes to the board.
10.5 Decisions
Board decisions in month five: fund MFA for the factory's remote vendor access, and approve the cost of ERP log collection. Each decision closes a gap Priya had reported. The factory vendor connected remotely without MFA, which was the same weakness that let the March attacker in through the VPN. ERP log collection closes the most important coverage gap. Priya presents each decision with its cost and the risk it reduces, so the board can weigh them, and records the board's decisions in the next monthly report.
10.6 Honesty
Reporting what went badly builds trust; the 26-minute provider test is reported with its fix. The 26-minute test missed the fifteen-minute target, and Priya reports it alongside the fix: the provider added a backup analyst. Hiding it would make the SOC look better but would leave the board unaware of a real weakness in night cover. Reporting problems with their fixes also shows the board that the SOC finds and corrects its own weaknesses, which supports the second-year budget request.
Audience
Board members are not security specialists. Priya avoids jargon, explains each chart in one sentence and links each risk to business impact: lost designs, stopped production or customer trust. She practises the presentation with the finance director, who tells her which terms were unclear. The board responds better to the story of March than to any chart, so she always begins with a short real example before the numbers.
Trends over time
Each chart shows at least six months so the board sees direction, not just a single month. When the median acknowledge time rose one month, Priya explained it was due to two holiday weeks with reduced provider staff, and the provider added cover for the next holiday period. Trends make such explanations visible and keep the board focused on lasting change rather than month-to-month noise.
Worked example
Six months after March, the board page shows: critical coverage 5/5, median time to acknowledge critical alerts 9 minutes, VPN rule precision 0.38, two of three tabletop findings closed. The board approves the second-year budget.
Synthetic case. Figures are invented.
Errors and limits
Board metrics summarise; they cannot show everything; keep detail available.
Exercises
- EX046 — Metric list. List Priya's five metrics. Rubric: 3 = coverage, acknowledge time, precision, open risks, findings closed; 2 = four; 1 = two; 0 = none.
- EX047 — Excluded metric. Which metric was left out and why? Rubric: 3 = alerts handled; grows with noise; 2 = partial; 1 = vague; 0 = none.
- EX048 — Board decisions. What did the board decide in month five? Rubric: 3 = MFA for vendor access; ERP log cost; 2 = one; 1 = vague; 0 = none.
- EX049 — Six-month figures. Give two six-month figures. Rubric: 3 = coverage 5/5; 9 min; precision 0.38; 2/3 findings (two); 2 = one; 1 = vague; 0 = none.
- EX050 — Honest reporting. Why report the failed test? Rubric: 3 = builds trust; shows fix; 2 = partial; 1 = vague; 0 = none.
Synthesis
Report a few honest metrics that lead to decisions.
Continue studying — chapter 10 of 12
← Chapter 09 (draft) · Course chapter index · Chapter 11 (draft) →