Chapter 46 — Engineering capstone II
Lesson map
Lesson 46 published. Map, example, EX136.
flowchart TD
N0["Learning outcomes"]
N1["46.1 Running the pilot"]
N2["46.2 Analysing results"]
N3["Lab — Exact test comparing contractor and employee r"]
N4["46.3 Scaling decision"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX136"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["46.1 Running the pilot"]
N2["46.2 Analysing results"]
N3["Lab — Exact test comparing contractor and employee r"]
N4["46.3 Scaling decision"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX136"]
[Learning outcomes]
|
v
[46.1 Running the pilot]
|
v
[46.2 Analysing results]
|
v
[Lab — Exact test comparing contractor and employee r]
|
v
[46.3 Scaling decision]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX136]
Learning outcomes
You will (1) pilot the capstone design; (2) analyse small-group results with an exact test; (3) decide whether to scale up.
46.1 Running the pilot
Aiko pilots the contractor design with sixty contractors over three months. They receive the induction directly from Greyford, can report through the web form or forwarding address and follow the same payment and access verification rules as employees. She tracks reports of real malicious emails, verdict times and burden scores. Two contractors leave and three join during the pilot, which she records. The pilot includes one month of announced simulations under the existing ethical rules, with contractor representatives on the approval panel for templates.
46.2 Analysing results
During the simulation month, contractors reported 21 of 60 simulated emails and employees in comparable roles reported 26 of 60. The lab computes Fisher's exact test, suitable for small groups, and finds no clear evidence of a difference. Aiko reports both rates with the test result and notes that the pilot was too small to detect modest differences. Contractors' burden scores are similar to employees', and the median verdict time for their reports is within the one-hour service level, meeting the main success criteria set in advance.
Lab — Exact test comparing contractor and employee reporting
# Fisher's exact test (two-sided) for contractor vs employee reporting in simulations (synthetic).
from math import comb
a, n1, b, n2 = 21, 60, 26, 60
K, N = a + b, n1 + n2
def p(x): return comb(n1, x) * comb(n2, K - x) / comb(N, K)
p0 = p(a)
pv = sum(p(x) for x in range(max(0, K - n2), min(K, n1) + 1) if p(x) <= p0 * (1 + 1e-9))
print(f"contractors {a}/{n1} = {a / n1:.0%}, employees {b}/{n2} = {b / n2:.0%}, two-sided p = {pv:.2f}")
Run with python3 lab_fisher.py. Verified output:
contractors 21/60 = 35%, employees 26/60 = 43%, two-sided p = 0.45
46.3 Scaling decision
The stopping rule defined in chapter 44 would have triggered if contractor reporting fell below half the employee rate; it did not. Aiko recommends scaling to all contractors, with the residual risk about personal devices reviewed after another three months. The board approves. She notes that the decision rests on meeting predefined criteria rather than on a significant result, which is appropriate because the aim was parity, not superiority. Planning the decision rule in advance prevented her from interpreting the results to suit her hopes.
Worked example
Aiko pilots contractor reporting with sixty contractors, finds no clear difference from employees with Fisher's exact test and scales up under predefined criteria.
Synthetic case. Pilot data are synthetic.
Errors and limits
Small pilots cannot show equivalence precisely. Contractor turnover affects results.
Exercises
- EX136 — Pilot rates. What were the simulation reporting rates? Rubric: 3 = contractors 21/60 and employees 26/60;; 2 = 35% vs 43%;; 1 = vague;; 0 = none.
- EX137 — Lab test. What p-value did Fisher's exact test give? Rubric: 3 = about 0.45;; 2 = not significant;; 1 = 0.05;; 0 = none.
- EX138 — Decision basis. What was the scaling decision based on? Rubric: 3 = predefined criteria including the stopping rule;; 2 = criteria;; 1 = significance;; 0 = none.
Synthesis
Pilots judged against predefined criteria support honest decisions even when samples are small.
Continue studying — chapter 46 of 50
← Chapter 45 (draft) · Course chapter index · Chapter 47 (draft) →