Chapter 6 — Security architecture and engineering: controlled evaluation
Lesson map
Lesson 06 published. Map, example, EX026.
flowchart TD
N0["Learning outcomes"]
N1["6.1 Designing the pilot"]
N2["6.2 Measuring"]
N3["6.3 Interpreting carefully"]
N4["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4
N4 --> EX["Worked example"]
EX --> Q["Exercise EX026"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["6.1 Designing the pilot"]
N2["6.2 Measuring"]
N3["6.3 Interpreting carefully"]
N4["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX026"]
[Learning outcomes]
|
v
[6.1 Designing the pilot]
|
v
[6.2 Measuring]
|
v
[6.3 Interpreting carefully]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX026]
Learning outcomes
You will (1) design a controlled pilot of the hybrid design; (2) measure outcomes against a comparison group; (3) interpret limited results carefully.
6.1 Designing the pilot
Before rolling the hybrid design across twenty-two clinics, Priya runs a pilot in six clinics, chosen by lot from clinics with similar size and connectivity, while six others continue as before. She defines outcomes in advance: minutes of access outage per clinic per month, staff sign-in time and help desk tickets about access. Random choice of pilot clinics prevents her from picking the easiest sites. Wohlin and colleagues (2012) describe this kind of controlled comparison and the threats to validity it must address.
6.2 Measuring
Over three months, pilot clinics record an average of eleven outage minutes a month against thirty-four in comparison clinics, mainly because certificate expiry outages disappear. Sign-in time rises by four seconds because of an extra identity step. Help desk tickets rise in the first month and then fall below the comparison level. Priya reports each outcome with the number of clinics and months behind it, so that readers can see how small the sample is. She also logs one fallback test in each pilot clinic.
6.3 Interpreting carefully
Six clinics over three months is a small sample. Priya writes that the results are consistent with better availability and acceptable staff impact, but that a single bad month could change the picture. She recommends a staged rollout with continued measurement rather than a big-bang change. The board agrees. Dr Voss highlights that pre-defined outcomes, random choice and honest sample reporting are what make a pilot persuasive, more than the size of the improvement itself. The full monthly figures are published in an appendix for anyone who wants to check them.
Worked example
Priya pilots the hybrid design in six randomly chosen clinics, measures three pre-defined outcomes against six comparison clinics and recommends a staged rollout.
Synthetic case. Clinics and figures are synthetic; Wohlin et al. (2012) is real.
Errors and limits
Small samples are fragile. Staff may behave differently when they know they are in a pilot.
Exercises
- EX026 — Choosing by lot. Why choose pilot clinics by lot? Rubric: 3 = to prevent picking the easiest sites;; 2 = bias;; 1 = convenience;; 0 = none.
- EX027 — Outage result. What were monthly outage minutes in pilot and comparison clinics? Rubric: 3 = eleven versus thirty-four;; 2 = lower;; 1 = vague;; 0 = none.
- EX028 — Staff cost. What staff cost did the pilot show? Rubric: 3 = sign-in rose by four seconds;; 2 = four seconds;; 1 = vague;; 0 = none.
- EX029 — Ticket pattern. How did help desk tickets change? Rubric: 3 = rose in month one, then fell below comparison;; 2 = rise then fall;; 1 = flat;; 0 = none.
- EX030 — Recommendation. What did Priya recommend after the pilot? Rubric: 3 = a staged rollout with continued measurement;; 2 = staged;; 1 = big bang;; 0 = none.
Synthesis
A controlled pilot with pre-defined outcomes gives modest but credible evidence for a staged decision.
Continue studying — chapter 06 of 12
← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →