Chapter 4 — Baseline replication and pilot
Lesson map
Lesson 04 published. Map, example, EX021.
flowchart TD
N0["Learning outcomes"]
N1["4.1 Why replicate first"]
N2["4.2 Baseline result"]
N3["4.3 Pilot goals"]
N4["4.4 Testing the pipeline"]
N5["4.5 Revising the protocol"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX021"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["4.1 Why replicate first"]
N2["4.2 Baseline result"]
N3["4.3 Pilot goals"]
N4["4.4 Testing the pipeline"]
N5["4.5 Revising the protocol"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX021"]
[Learning outcomes]
|
v
[4.1 Why replicate first]
|
v
[4.2 Baseline result]
|
v
[4.3 Pilot goals]
|
v
[4.4 Testing the pipeline]
|
v
[4.5 Revising the protocol]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX021]
Learning outcomes
You will (1) replicate a published or prior baseline before extending it; (2) run a pilot to test procedures, not effects; (3) check data pipelines against known values; (4) revise the protocol from pilot evidence.
4.1 Why replicate first
Before testing her intervention, Amara checks that she can reproduce Tessaro's own historical baseline: about four architecture defects per hundred design changes. If her extraction and coding produce a very different number from the organisation's records, either the records or her method is wrong, and any later comparison would be built on sand. Replication of the baseline is a cheap test of the whole measurement chain. She extracts two years of historical changes and defects, codes them with the manual from chapter 1 and compares her rate with the figure security leadership reported in planning documents.
4.2 Baseline result
Her coding gives 3.6 architecture defects per hundred changes, slightly below the reported 4.1. Investigating, she finds that the reported figure counted duplicate tickets for the same defect when several teams were affected. After deduplicating, both numbers agree within rounding. This small discovery matters: had she compared her later measurements with the inflated planning figure, the intervention would have appeared to work even if nothing changed. She records the discrepancy and the fix in her change log and informs Tessaro, whose planning team corrects the dashboard for future reporting.
4.3 Pilot goals
The pilot runs with three volunteer teams for three months. Its goals are procedural: can teams complete the protocol in ninety minutes, can the fidelity checklist be scored consistently, does the defect data pipeline capture every relevant ticket, and do engineers find the session acceptable. The pilot is explicitly not meant to estimate the effect, because three teams over three months would produce too few defects for any conclusion. Amara states this in the pilot plan so nobody, including herself, is tempted to announce early success from a handful of numbers.
4.4 Testing the pipeline
Amara inserts ten synthetic test tickets with known classifications into a copy of the defect tracker and runs her extraction scripts. Nine arrive correctly; one, filed under a renamed project, is missed. She fixes the project mapping and reruns, recovering all ten. She keeps the synthetic tickets as a permanent regression test that runs before each data extraction. This practice, borrowed from software testing, gives a known-answer check for the research pipeline in the same way that test vectors give a known-answer check for cryptographic code.
4.5 Revising the protocol
Pilot evidence leads to three revisions. Sessions for large changes ran long, so the protocol now allows splitting them into two sessions. Fidelity scoring disagreed on whether trust boundaries were drawn, so the checklist gains an example diagram. Engineers found the external architect's review useful but slow, so reviews must now finish within five working days. Each revision is logged with its reason before the main study starts. Amara reports the pilot in the thesis as a separate chapter, including the problems, because future replicators will face the same practical difficulties.
Worked example
Amara replicates the baseline at 3.6 per hundred changes, traces the 4.1 figure to duplicate tickets, fixes a pipeline gap with synthetic test tickets and logs three protocol revisions.
Synthetic case. Baseline figures, pilot teams and tickets are synthetic.
Errors and limits
A pilot with volunteer teams may be more motivated than typical teams. Regression tests catch known failure modes only.
Exercises
- EX021 — Baseline discrepancy. What explained the gap between 3.6 and 4.1? Rubric: 3 = duplicate tickets for the same defect across teams in the reported figure;; 2 = duplicates;; 1 = coding error;; 0 = none.
- EX022 — Inflated comparison risk. What would have happened if the inflated figure were used as baseline? Rubric: 3 = the intervention would appear to work even without real change;; 2 = false success;; 1 = vague;; 0 = none.
- EX023 — Pilot not effect. Why was the pilot not used to estimate the effect? Rubric: 3 = three teams over three months yield too few defects;; 2 = too few;; 1 = it was;; 0 = none.
- EX024 — Missed ticket. Why was one synthetic ticket missed, and how was it fixed? Rubric: 3 = filed under a renamed project; fixed the project mapping;; 2 = cause;; 1 = vague;; 0 = none.
- EX025 — Test ticket analogy. What do the synthetic tickets resemble in cryptographic practice? Rubric: 3 = known-answer test vectors;; 2 = test vectors;; 1 = vague;; 0 = none.
- EX026 — Large changes. How was the protocol revised for large changes? Rubric: 3 = allowed splitting into two sessions;; 2 = split;; 1 = shortened;; 0 = none.
Synthesis
Replicating the baseline and piloting procedures exposes measurement errors before they can masquerade as effects.
Continue studying — chapter 04 of 08
← Chapter 03 (draft) · Course chapter index · Chapter 05 (draft) →