Chapter 11 — Applied project
Lesson map
Lesson 11 published. Map, example, EX051.
flowchart TD
N0["Learning outcomes and connection"]
N1["11.1 Choosing the project"]
N2["11.2 The project charter"]
N3["11.3 Delivering through people"]
N4["11.4 Evaluating the result"]
N5["11.5 Reporting the project"]
N6["11.6 Assessment rubric"]
N7["11.8 Practical activity: did it work?"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX051"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["11.1 Choosing the project"]
N2["11.2 The project charter"]
N3["11.3 Delivering through people"]
N4["11.4 Evaluating the result"]
N5["11.5 Reporting the project"]
N6["11.6 Assessment rubric"]
N7["11.8 Practical activity: did it work?"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX051"]
[Learning outcomes and connection]
|
v
[11.1 Choosing the project]
|
v
[11.2 The project charter]
|
v
[11.3 Delivering through people]
|
v
[11.4 Evaluating the result]
|
v
[11.5 Reporting the project]
|
v
[11.6 Assessment rubric]
|
v
[11.8 Practical activity: did it work?]
|
v
[Worked example] --> [Exercise EX051]
Learning outcomes and connection
After this chapter you should be able to:
- Scope an organization-scale security improvement as a project with a decision-maker, a consequence-based objective and measurable benefits.
- Plan delivery with stakeholders, risks, safety conditions and checkpoints.
- Evaluate the result honestly with a comparison group, a difference-in-differences estimate and a permutation check.
- Test the robustness of a project result and report its limits.
- Present the project for assessment against an explicit rubric.
The applied project is where the course becomes practice. It uses every earlier chapter: framing and decision records (1), current practice and verification (2), evidence (3), risk and ethics (4), design (5), controlled evaluation (6), integration (7), difficult-case thinking (8), assurance (9) and critical reading (10). Unlike a master's dissertation, its purpose is not to answer a research question for others; it is to improve a real service and show credibly whether it did. Chapter 12 then reviews the course and transfers it.
11.1 Choosing the project
A good postgraduate applied project is: consequential (it changes an essential service's exposure, not just a document); bounded (deliverable within about two semesters of part-time work); owned (a named sponsor with authority, and operational owners who want it); measurable (a benefit that can be observed, preferably in service units); safe and authorized (it creates no unapproved risk to operations, people or data); and evaluable (there is a way to compare with what would have happened without it). Typical projects: manual-operation readiness for a group of sites; brokered vendor access for one control system; a restore capability for clinical systems; a regional supplier-concentration plan. If you are not employed in the sector, the project can be carried out on a realistic synthetic case like Merrow Bay, with the same deliverables.
11.2 The project charter
A one-page charter fixes: the problem and its consequence; the sponsor and decision rights; objectives and benefits with measures and baselines; scope and exclusions; deliverables; stakeholders; constraints (safety, budget, regulation); risks; the evaluation design; milestones and checkpoints; and the assessment rubric. Writing the evaluation design into the charter before delivery is the project equivalent of pre-registration (Nosek et al., 2018): it protects the specialist from the temptation to choose, afterwards, the measure that looks best (Kerr, 1998).
11.3 Delivering through people
Most security projects in essential services fail on adoption, not technology. Map stakeholders by influence and interest; agree with operations how and when changes touch plant; plan training and supervised practice; and anticipate who loses something (convenience, autonomy, budget). A pre-mortem at the start (Klein, 2007) surfaces adoption risks early. Keep a project decision log in the Chapter 1 format so later reviewers can see why the plan changed.
At Merrow Water the people most affected by the runbook project are shift operators, who must practise manual operation during quarterly drills on top of normal work. Agreeing drill times with shift managers and using their feedback to revise the runbook makes adoption far more likely than issuing it from head office.
11.4 Evaluating the result
Before-and-after comparisons are seductive and weak: the outcome may have improved anyway because of season, practice, new staff or other projects. A stronger, practical design is difference-in-differences (DiD): compare the change at sites that received the project (pilot) with the change at similar sites that did not (comparison) over the same period. The estimate is (pilot after − pilot before) − (comparison after − comparison before). Its key assumption is parallel trends: without the project, pilot and comparison sites would have changed by the same amount. The design was popularized in economics by a well-known study comparing employment in neighbouring states (Card and Krueger, 1994); it applies equally to drill times or recovery times across sites.
With few sites, ordinary standard errors are unreliable. A permutation check is simple and honest: compute the DiD for every possible way of choosing which sites were "pilot", and ask how unusual the real assignment's estimate is. If many random assignments produce estimates as large, the result could be chance. Finally, test robustness: drop the most influential site, use a different outcome definition, or check whether pilot and comparison sites were already diverging before the project.
11.5 Reporting the project
The final report states the problem, what was done, what changed, how sure we are, and what remains. It reports the pre-specified measure first; additional analyses are labelled exploratory. Limitations are written with their direction — would this weakness make the effect look larger or smaller? — so readers can judge the result. Benefits that could not be measured are described, not invented. The report ends with handover: owners, maintenance, the assurance evidence that will keep the claim fresh (Chapter 9) and the review trigger.
11.6 Assessment rubric
Projects are assessed against five criteria, each scored 0–3:
| Criterion | 3 (excellent) | 1 (weak) |
|---|---|---|
| Framing | Consequence-based problem, sponsor, measurable benefit, evaluation pre-specified | Problem vague; measure chosen after results |
| Delivery | Safe, authorized, adopted by operations; decisions logged | Delivered to paper only; operations bypassed |
| Evaluation | Comparison group, DiD or equivalent, uncertainty and robustness reported | Before-and-after only, no uncertainty |
| Ethics and safety | Risks to people and service assessed; distribution considered | Not considered |
| Communication and handover | Clear report, directed limitations, owners and assurance evidence | Advocacy; no handover |
11.7 Worked example: the manual-operation runbook project (synthetic)
Merrow Water's specialist ran a project to introduce a manual-operation runbook and quarterly drills at six works (pilot), chosen because they had the oldest control systems; six other works continued as before (comparison). The pre-specified outcome was mean minutes to stable manual operation in quarterly drills, averaged over two quarters before and two after. The charter pre-specified DiD with a permutation check and a robustness check dropping the most influential site.
Choosing pilot sites for their old control systems was sensible for the service, because those works most needed the runbook, but it means the pilot and comparison groups were not alike at the start. That is why the charter required difference-in-differences rather than an after-only comparison, and why parallel trends must be argued rather than assumed. The six comparison works are expected to receive the runbook in the next phase, which gives a later chance to test the result.
11.8 Practical activity: did it work?
# Lab 11 - did the project work? Difference-in-differences with a permutation check (offline, stdlib only)
# Synthetic: Merrow Water piloted a manual-operation runbook and drills at six works (pilot) and not at six others.
# Outcome: mean minutes to stable manual operation in quarterly drills, two quarters before and two after.
from itertools import combinations
SITES = { # site: (pilot?, mean minutes before, mean minutes after)
"Hollin": (1, 41, 27), "Fenley": (1, 38, 29), "Upper Merrow": (1, 55, 36), "Kettle Ford": (1, 33, 28),
"Ashby": (1, 47, 35), "Saltings": (1, 36, 30),
"Brack": (0, 39, 35), "Dunmere": (0, 44, 40), "Orrin": (0, 35, 34), "Pelling": (0, 50, 43),
"Quarry Lane": (0, 31, 30), "Rook": (0, 42, 36),
}
def did(treated):
t = [SITES[s][2] - SITES[s][1] for s in SITES if s in treated]
c = [SITES[s][2] - SITES[s][1] for s in SITES if s not in treated]
return sum(t) / len(t) - sum(c) / len(c), sum(t) / len(t), sum(c) / len(c)
pilot = {s for s, v in SITES.items() if v[0]}
est, dt, dc = did(pilot)
print(f"pilot change {dt:+.2f} min, comparison change {dc:+.2f} min, difference-in-differences {est:+.2f} min")
naive = sum(SITES[s][2] for s in pilot) / 6 - sum(SITES[s][2] for s in SITES if s not in pilot) / 6
print(f"naive after-only difference {naive:+.2f} min")
allc = [did(set(c))[0] for c in combinations(SITES, 6)]
p = sum(1 for d in allc if d <= est) / len(allc)
print(f"permutation: {len(allc)} assignments; share with DiD <= observed = {p:.4f}")
# Sensitivity: drop the site with the largest pilot improvement
big = min(pilot, key=lambda s: SITES[s][2] - SITES[s][1])
rest = {s: SITES[s] for s in SITES if s != big}
t = [v[2]-v[1] for s, v in rest.items() if v[0]]; c = [v[2]-v[1] for s, v in rest.items() if not v[0]]
print(f"without {big}: DiD {sum(t)/len(t) - sum(c)/len(c):+.2f} min")
Expected output:
pilot change -10.83 min, comparison change -3.83 min, difference-in-differences -7.00 min
naive after-only difference -5.50 min
permutation: 924 assignments; share with DiD <= observed = 0.0087
without Upper Merrow: DiD -5.37 min
Interpretation. Pilot sites improved by 10.8 minutes on average; comparison sites by 3.8 minutes, probably through general practice and awareness. The DiD estimate is a 7.0-minute improvement attributable to the project under the parallel-trends assumption. A naive after-only comparison (5.5 minutes) would understate it, because pilot sites started slower — they were chosen for their old systems. Of 924 possible assignments of six "pilot" sites, fewer than 1% produce an improvement as large as the real one, so chance is an unlikely explanation. Dropping Upper Merrow, the site with the largest improvement, reduces the estimate to 5.4 minutes: the result is robust in direction but sensitive in size to one site. The report should state "about 5 to 7 minutes faster, mainly but not only driven by Upper Merrow", note that pilot sites were selected for old systems (so parallel trends is an assumption to defend), and recommend extending the runbook to comparison sites with the same evaluation.
11.9 Common errors and limits
- Choosing a project for its interest rather than its consequence.
- Starting without a sponsor who can make decisions.
- Defining the success measure after seeing results.
- Evaluating with before-and-after only.
- Reporting a single estimate without a chance check or robustness test.
- Listing limitations without their direction.
- Finishing without handover, so the benefit decays.
With a dozen sites, no evaluation is definitive. The project's evaluation should be honest about that and designed so the next phase (for example, rolling out to comparison sites in a staggered order) yields more evidence.
Exercises — chapter 11
EX051 — Project charter. Objective: scope your own applied project. Inputs: your organization, or Merrow Bay. Constraints: one page; all charter elements in §11.2; evaluation pre-specified. Steps: write the charter. Expected evidence: charter. Rubric: use §11.6 "Framing" and "Ethics and safety" criteria; 3 = consequential, bounded, owned, measurable, safe and evaluable, with a named comparison; 2 = evaluation vague; 1 = activity rather than outcome; 0 = none.
EX052 — Robustness of the permutation result. Objective: check sensitivity to one site. Inputs: Lab 11. Constraints: remove Upper Merrow from the data entirely (11 sites, 5 pilot). Steps: (1) recompute DiD; (2) recompute the permutation share over all assignments of 5 pilots among 11 sites; (3) interpret. Expected evidence: numbers and paragraph. Rubric: 3 = DiD −5.37 minutes, 462 assignments, share about 0.017; interpretation that the result remains unlikely under chance but is weaker, with direction of the selection limitation; 2 = numbers right, interpretation thin; 1 = not run; 0 = none.
EX053 — Stakeholder and adoption plan. Objective: plan delivery through people. Inputs: the runbook project. Constraints: include night shifts and contractors. Steps: map stakeholders; identify who loses what; plan training, supervised practice and feedback; run a short pre-mortem and list three failure stories with mitigations. Expected evidence: plan. Rubric: 3 = stakeholders specific, losses acknowledged, practice-based training, credible pre-mortem stories; 2 = no pre-mortem; 1 = communications plan only; 0 = none.
EX054 — Directed limitations. Objective: write limitations that help readers judge. Inputs: Lab 11 result. Constraints: at least five limitations, each with direction and how a later phase could address it. Steps: write them. Expected evidence: list. Rubric: 3 = selection of pilot sites, regression to the mean (slow sites improve more anyway — would inflate), practice effects, small sample, drill realism versus real events, contamination of comparison sites; directions correct; 2 = directions missing; 1 = generic; 0 = none.
EX055 — Handover and benefits. Objective: make benefits last. Inputs: the runbook project at completion. Constraints: use Chapter 9's assurance case and freshness ideas. Steps: (1) name owners; (2) define the assurance claim and evidence with maximum ages; (3) define an indicator for the board with an XmR baseline; (4) define the review trigger. Expected evidence: handover note. Rubric: 3 = operational owners, verified-grade evidence with ages, indicator with baseline period stated, trigger specific; 2 = one element missing; 1 = "project closed"; 0 = none.
Chapter synthesis
An applied project improves a real service and shows credibly whether it did. Choose something consequential, bounded, owned, measurable, safe and evaluable; fix the evaluation in a charter before delivery; deliver through people, with operations' consent and a pre-mortem; evaluate with a comparison group, difference-in-differences, a permutation check and robustness tests; report limitations with their direction; and hand over with owners, assurance evidence and a review trigger.
Continue studying — chapter 11 of 12
← Chapter 10 · Course chapter index · Chapter 12 →
Continue to the next authored lesson.