Chapter 23 — Security architecture and engineering: evaluation | Technologist Security architecture and engineering

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 23 — Security architecture and engineering: evaluation

Lesson map

Lesson 23 published. Map, example, EX067.

flowchart TD
  N0["Learning outcomes"]
  N1["23.1 Did it work"]
  N2["23.2 Before and after"]
  N3["Synthesis"]
  N0 --> N1 --> N2 --> N3
  N3 --> EX["Worked example"]
  EX --> Q["Exercise EX067"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["23.1 Did it work"]
  N2["23.2 Before and after"]
  N3["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> Q["EX067"]

[Learning outcomes]
    |
    v
[23.1 Did it work]
    |
    v
[23.2 Before and after]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX067]

Learning outcomes

You will (1) evaluate whether controls achieved their goals; (2) compare before and after figures; (3) report limits of the evaluation.

23.1 Did it work

Six months into the placement, Sam asks whether the changes made a difference. Tomás returns to the health measures from chapter 9 and the risk scenarios from chapter 13. For each control he asks: was it implemented, is it operating, and did the risk change? He uses evidence rather than impressions. The upload portal now carries ninety percent of manifests; emailed spreadsheets have fallen to a handful from very small retailers. The manifest check has flagged two suspicious uploads, both confirmed as fraud attempts by the retailers.

23.2 Before and after

Tomás compares figures. Devices updated within thirty days rose from forty-eight to eighty-five percent. Average days to disable leavers fell from four to one. Inbound internet rules fell from two to zero. Unowned devices fell from three to zero. He presents each with its source and notes that some improvement may come from other causes, such as the vendor releasing an easier scanner update tool. Before-and-after figures are persuasive but do not by themselves prove which change caused the improvement.

23.3 Limits

Tomás lists what the evaluation cannot show. No serious incident in six months may reflect luck as much as controls. Two fraud attempts caught does not reveal how many were missed. Staff behaviour may slip once the placement ends. He recommends continuing the measures monthly and repeating the evaluation in a year. Ruth praises the honesty, noting that an evaluation which admits its limits is more useful to managers than one that claims complete success, because it tells them what to keep watching.

Worked example

Tomás evaluates controls with before-and-after measures, records two caught fraud attempts and lists the limits of what the evaluation shows.

Synthetic case. Figures are synthetic.

Errors and limits

Absence of incidents is weak evidence. Attribution of improvement is uncertain.

Exercises

EX067 — Portal share. What share of manifests now uses the portal? Rubric: 3 = ninety percent;; 2 = 90%;; 1 = all;; 0 = none.
EX068 — Update improvement. How did the thirty-day update share change? Rubric: 3 = from 48 to 85 percent;; 2 = rise;; 1 = vague;; 0 = none.
EX069 — No incidents. Why is the absence of serious incidents weak evidence? Rubric: 3 = it may reflect luck as much as controls;; 2 = luck;; 1 = vague;; 0 = none.

Synthesis

Honest evaluation compares measured outcomes and states clearly what the evidence cannot prove.

Continue studying — chapter 23 of 30

← Chapter 22 (draft) · Course chapter index · Chapter 24 (draft) →