Chapter 23 — Evaluate response latency and decision quality
Lesson map
Lesson 23 published. Map, example, EX067.
flowchart TD
N0["Learning outcomes and connection"]
N1["23.1 Stage latency"]
N2["23.2 Decision quality"]
N3["23.3 Lab T-23 synthetic : six cases"]
N4["23.4 From evaluation to change"]
N5["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX067"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["23.1 Stage latency"]
N2["23.2 Decision quality"]
N3["23.3 Lab T-23 synthetic : six cases"]
N4["23.4 From evaluation to change"]
N5["Chapter synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX067"]
[Learning outcomes and connection]
|
v
[23.1 Stage latency]
|
v
[23.2 Decision quality]
|
v
[23.3 Lab T-23 synthetic : six cases]
|
v
[23.4 From evaluation to change]
|
v
[Chapter synthesis]
|
v
[Worked example] --> [Exercise EX067]
Learning outcomes and connection
You will be able to (1) break response time into stages and find where delay accumulates; (2) evaluate decision quality from the record rather than from outcomes; and (3) combine both into a service review that leads to specific changes. Chapter 9 defined measures; Chapters 17–22 produced the records; now you evaluate the service.
23.1 Stage latency
Total response time hides where delay lives. Split it: detect → triage (queue and on-call), triage → decision (information gathering and authority), decision → containment (execution and change path). Each stage has different causes and fixes: paging for the first, evidence access and pre-approval for the second, automation and tested actions for the third. Fernmoor's charter maps onto this split: paging and the on-call rota govern the first stage, the list of pre-approved actions and owner approvals governs the second, and the tested action cards govern the third. A delay in each stage therefore points to a different owner.
23.2 Decision quality
A good outcome does not prove a good decision, and a bad outcome does not prove a bad one; judging decisions by outcomes is hindsight bias (Fischhoff, 1975). Evaluate decisions by what the record shows was known and done at the time: was there a named owner, were the key facts gathered, were options considered, was a rollback or revisit condition stated? Fernmoor's decision record requires four elements: owner, evidence relied on, option chosen with alternatives, and rollback or revisit condition.
23.3 Lab T-23 (synthetic): six cases
# Lab T-23: stage latency and decision-quality checks for six synthetic cases
from statistics import median
# minutes: detect->triage, triage->decision, decision->contain; decision record fields present
cases = {"FCS-301":(9,25,14,{"owner","evidence","option","rollback"}),
"FCS-302":(31,12,6,{"owner","option"}),
"FCS-303":(7,48,22,{"owner","evidence","option","rollback"}),
"FCS-304":(12,15,95,{"owner","evidence","option"}),
"FCS-305":(5,20,11,{"owner","evidence","option","rollback"}),
"FCS-306":(64,18,9,{"evidence","option"})}
need = {"owner","evidence","option","rollback"}
for i,name in enumerate(["detect->triage","triage->decision","decision->contain"]):
v=[c[i] for c in cases.values()]
print(f"{name:18} median {median(v):5.1f} max {max(v)}")
tot={k:sum(c[:3]) for k,c in cases.items()}
print("total median", median(tot.values()))
for k,c in cases.items():
miss=sorted(need-c[3])
if miss: print(k, "total", tot[k], "missing", miss)
q=sum(1 for c in cases.values() if need<=c[3])
print("complete decision records", q, "of", len(cases))
Expected output:
detect->triage median 10.5 max 64
triage->decision median 19.0 max 48
decision->contain median 12.5 max 95
total median 63.0
FCS-302 total 49 missing ['evidence', 'rollback']
FCS-304 total 122 missing ['rollback']
FCS-306 total 91 missing ['owner', 'rollback']
complete decision records 3 of 6
Interpretation: the medians look acceptable, but the maxima reveal three different problems. FCS-306 waited 64 minutes for triage and had no recorded owner — a paging and ownership issue. FCS-304 took 95 minutes from decision to containment and lacked a rollback — consistent with an untested, unprepared action (it was a firewall change without a pre-built procedure). FCS-302 decided quickly but without recorded evidence or rollback — speed bought with poor decision hygiene. Only half the decision records were complete. These are records of six cases; they suggest where to look, not statistically established causes.
23.4 From evaluation to change
For each finding, propose one change, an owner and a measure to check it: build and test the firewall containment action (target: decision → contain ≤ 30 minutes for that action); make owner and rollback required fields at decision status (target: 100% completeness); review on-call escalation for S2 out of hours. Re-evaluate after a quarter with the same definitions. Each change is tested against the same measure that revealed the problem, so a quarter later the team can see whether the stage got faster rather than relying on impressions. If it did not, the change is reconsidered instead of left in place.
23.5 Common errors and limits
- Judging decisions by outcomes.
- Only reporting totals or means.
- Treating six cases as proof.
- Measuring individuals instead of the system; people will game the fields.
Fernmoor's evaluation was written with these in mind. A case that ended well despite a poor decision was still recorded as a decision finding; stage medians and percentiles replaced a single mean; conclusions from a handful of cases were labelled as tentative; and figures were reported by service stage, not by analyst. Analysts who expect blame learn to fill fields defensively.
Exercises — chapter 23
Attempt each exercise before opening the worked results. Every case, organization, person, address and number in an exercise is synthetic unless a cited public source is named.
EX067 — Recompute. Steps: add rollback to FCS-304 and owner to FCS-306 in the lab; what changes? Rubric: 3 = complete records 4 of 6, FCS-304 no longer listed, FCS-306 lists only rollback; latencies unchanged; 2 = partly right; 1 = attempted; 0 = none.
EX068 — Decision review. Steps: review FCS-302's decision (disabled a public form within 6 minutes on an analyst's hunch; it turned out to be a real attack). Was it a good decision? Rubric: 3 = separates outcome from process; notes missing evidence and rollback; acknowledges speed was appropriate if authority existed; recommends recording; 2 = judges by outcome only partly; 1 = "good because it worked"; 0 = none.
EX069 — Change proposal. Steps: write one change proposal for FCS-304's finding. Rubric: 3 = problem with data, change, owner, measure, review date; 2 = missing measure; 1 = vague; 0 = none.
Chapter synthesis
Split latency into stages to find where delay lives, evaluate decisions by the recorded process rather than outcomes, keep sample sizes in view, and convert each finding into an owned, measurable change.
Continue studying — chapter 23 of 30
← Chapter 22 · Course chapter index · Chapter 24 →
Continue to the next authored lesson.