OpenCyber AcademyLesson map
Lesson 05 published. Map, example, EX027.
flowchart TD
N0["Learning outcomes and connection"]
N1["5.1 The proposal as a contract"]
N2["5.2 Timeline and critical path"]
N3["5.3 The risk register"]
N4["5.4 What a pilot is for"]
N5["5.5 Progression criteria"]
N6["5.6 Practical activity: auditing ten pilot sessions"]
N7["5.8 Protocol deviations and instrument revisions"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX027"]
Logical drawing
flowchart LR
N0["Learning outcomes and connection"]
N1["5.1 The proposal as a contract"]
N2["5.2 Timeline and critical path"]
N3["5.3 The risk register"]
N4["5.4 What a pilot is for"]
N5["5.5 Progression criteria"]
N6["5.6 Practical activity: auditing ten pilot sessions"]
N7["5.8 Protocol deviations and instrument revisions"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX027"]
[Learning outcomes and connection]
|
v
[5.1 The proposal as a contract]
|
v
[5.2 Timeline and critical path]
|
v
[5.3 The risk register]
|
v
[5.4 What a pilot is for]
|
v
[5.5 Progression criteria]
|
v
[5.6 Practical activity: auditing ten pilot sessions]
|
v
[5.8 Protocol deviations and instrument revisions]
|
v
[Worked example] --> [Exercise EX027]
Learning outcomes and connection
After this chapter you should be able to:
- Write a research proposal that works as a contract between student, supervisor, ethics committee and partners: what will be done, how it will be judged and what happens if it cannot be done.
- Build a timeline around its critical path and a risk register with owners and triggers.
- Design a feasibility pilot whose objectives are recruitment, procedure and measurement — not the effect.
- Predeclare progression criteria with go, amend and stop thresholds, and audit pilot data against them.
- Record protocol deviations and instrument revisions with an audit trail, and state the stop rules for the main study.
The chapter builds on the design (Chapter 3) and rival table (Chapter 4) and prepares Chapter 6, where the revised instruments collect the main-study data.
5.1 The proposal as a contract
A master's proposal is often treated as a formality before "the real work". It is better understood as a contract. It tells the supervisor what will be delivered and when; it tells the ethics committee exactly what participants will experience; it tells partner organizations what will be asked of them and what they will receive; and it tells examiners, later, what the student committed to before seeing data. A proposal that is vague in any of these directions will be renegotiated under pressure, usually at the expense of rigour.
A proposal for the Triage Rubric Study has these parts:
Lesson table; scroll horizontally to read all columns →
| Section | Content | Why it matters |
|---|---|---|
| Problem and question | The MRSS decision, RQ1–RQ3, estimand, claim scope | Fixes what success means |
| Background | Concept matrix summary, competing models, gap | Shows the question is not already answered |
| Design | Crossover tabletop; historical strand as context | Shows why this design answers this question |
| Participants and recruitment | Eligibility, neutral channels, target 24, allocation | Ethics and feasibility |
| Materials | 24 main cases plus pilot cases, rubric, observer form, debrief guide | Lets others replicate |
| Measures and analysis | Pre-analysis plan (Chapter 3) | Separates confirmation from exploration |
| Rivals | Rival table and checks (Chapter 4) | Shows what the design rules out and what it does not |
| Pilot | Objectives and progression criteria | Protects the main study |
| Ethics and data | Approval route, safeguards, data management plan | Protects participants |
| Timeline and risks | Critical path, risk register | Protects the schedule |
| Outputs | Dissertation, reproducibility package, partner briefing | Defines delivery |
5.2 Timeline and critical path
The critical path is the chain of tasks whose delay delays the whole project. In empirical master's projects it is rarely the analysis; it is approvals, recruitment and access. For the Triage Rubric Study:
Month 1-2 question, review, design (supervisor sign-off)
Month 3 ethics application submitted ── critical: approval 6-10 weeks
Month 3-4 case writing, panel reference ratings ── critical: panel availability
Month 5 pilot (10 sessions) ── depends on approval + cases
Month 6 pilot report, instrument revision, amendment if needed
Month 7-9 main-study recruitment and 24 sessions ── critical: recruitment rate
Month 9-10 debrief interviews, transcription
Month 10-12 analysis per pre-analysis plan
Month 12-16 writing, internal review, submission
Month 17-18 examination, corrections, partner briefing
Build slack after each critical task. If ethics approval takes ten weeks instead of six, the pilot moves, not the analysis plan.
Two of the three items marked critical, ethics approval and panel availability, fall before the month 5 pilot, which cannot start until both are done. The third, recruitment for the 24 main-study sessions in months 7–9, comes later; if it runs slow, the slack should come from the writing months rather than from cutting sessions below what the pre-analysis plan requires.
5.3 The risk register
Lesson table; scroll horizontally to read all columns →
| Risk | Likelihood | Impact | Trigger | Response | Owner |
|---|---|---|---|---|---|
| Recruitment below 24 | Medium | High | Fewer than 12 enrolled by end of month 8 | Add second channel; extend one month; report precision achieved | Student |
| Panel disagreement on reference severity | Medium | Medium | Any case without consensus after discussion | Revise vignette or drop case before main study; record | Supervisor |
| Ethics amendment needed after pilot | High | Medium | Any change to procedure affecting participants | Submit amendment; do not start main study until approved | Student |
| Observer unavailability | Low | Medium | Fewer than two observers for a session | Single observer with audio time stamps | Student |
| Case leakage between participants | Medium | High | Participant mentions specific case content to colleague | Ask participants not to discuss; record sessions dates; check performance by date | Student |
| Data loss | Low | High | Storage failure | Encrypted institutional storage with backup; raw exports read-only | Student |
A risk register is useful only if the triggers are observable and the responses decided in advance.
5.4 What a pilot is for
A pilot study is a small version of the main study run to find out whether the main study can be done as planned. Its objectives are feasibility and process: can participants be recruited at the expected rate? Do they complete sessions? Are the instructions understood? Are the cases clear? Do the observers' forms capture what is needed? Can two observers agree on when a decision was made? Is the session length acceptable? Thabane et al. (2010) set out these purposes and warn against the most common misuse: treating the pilot as a small trial and estimating the effect. Effect estimates from a pilot are extremely imprecise, and a pilot that "finds no effect" may lead a team to abandon a good intervention, while one that finds a large effect invites an under-sized main study. The CONSORT extension for pilot and feasibility trials (Eldridge et al. 2016) gives a reporting framework that makes these objectives explicit.
How large should a pilot be? There is no universal number; it depends on what the pilot must estimate. Hertzog (2008) discusses how precision of feasibility estimates grows with pilot size. Ten sessions are enough to see whether procedures break and whether observers agree on a reasonable number of timing pairs; they are not enough to estimate a completion rate precisely, and the pilot report says so.
5.4.1 What the Triage Rubric pilot measures
- Completion: proportion of booked participants who complete a session.
- Field completeness: proportion of required observer fields (two times and the severity rating) recorded.
- Timing agreement: proportion of decisions where the two observers' times are within 30 seconds.
- Case clarity: number of participants who flag each case as unclear.
- Session length: minutes per session, against a 90-minute limit agreed with the ethics committee.
The pilot uses its own six cases (C1–C6), not the main-study cases, so that main-study materials are not exposed and pilot participants can, if eligible, be excluded from the main study without wasting cases. It records ratings, but nobody computes a rubric effect from them.
5.5 Progression criteria
Progression criteria are thresholds, set before the pilot, that decide whether the main study proceeds as planned (go), proceeds after changes (amend) or does not proceed in its current form (stop). A traffic-light form is common: green above a threshold, amber between two, red below. The thresholds come from the main study's needs: if the main study needs 90% field completeness to compute its outcomes without heavy imputation, that is the green threshold.
Lesson table; scroll horizontally to read all columns →
| Criterion | Go (green) | Amend (amber) | Stop (red) |
|---|---|---|---|
| Completion | ≥ 0.80 | 0.60–0.79 | < 0.60 |
| Field completeness | ≥ 0.90 | 0.80–0.89 | < 0.80 |
| Timing agreement (±30 s) | ≥ 0.80 | 0.60–0.79 | < 0.60 |
| Unclear flags per case | ≤ 3 | 4–5 | > 5 (that case cannot be used unrevised) |
| Sessions over 90 minutes | none | 1–2 | > 2 |
"Stop" on a case-level criterion stops that case, not the study.
5.6 Practical activity: auditing ten pilot sessions
Lab 5 audits ten synthetic pilot sessions with incomplete observer notes against the criteria. One participant (P04) stopped after three cases. Save it as lab5_pilot.py.
# Lab 5 - audit ten pilot tabletop sessions against predeclared progression criteria (offline, stdlib only)
# Synthetic pilot: ten volunteer analysts (P01-P10), each triaging 6 fictional cases while two observers time
# the severity decision. None = field left blank on the observer sheet. No outcome comparison is made here.
SESSIONS = { # pid: (completed, minutes, [(case, o1_seconds, o2_seconds, severity_recorded, unclear_flag)])
"P01": (True, 74, [("C1", 210, 205, 3, 0), ("C2", 95, 100, 2, 0), ("C3", 330, 290, 4, 1), ("C4", 150, 152, 1, 0), ("C5", 260, 255, 3, 0), ("C6", 120, 118, 2, 0)]),
"P02": (True, 81, [("C1", 240, None, 3, 0), ("C2", 110, 112, 2, 0), ("C3", 410, 400, 3, 1), ("C4", 140, 141, 1, 0), ("C5", 300, 240, 4, 0), ("C6", 90, 95, 2, 0)]),
"P03": (True, 69, [("C1", 180, 176, 2, 0), ("C2", 85, 88, 2, 0), ("C3", 290, 300, 4, 0), ("C4", 160, 158, 1, 0), ("C5", 230, 228, 3, 0), ("C6", 100, 99, 1, 0)]),
"P04": (False, 35, [("C1", 260, 255, 3, 0), ("C2", 130, None, None, 0), ("C3", None, None, None, 1)]),
"P05": (True, 96, [("C1", 300, 290, 3, 0), ("C2", 140, 139, 2, 0), ("C3", 520, 470, 4, 1), ("C4", 170, 172, 1, 0), ("C5", 340, 335, 3, 1), ("C6", 150, 150, 2, 0)]),
"P06": (True, 77, [("C1", 200, 201, 3, 0), ("C2", 100, None, 2, 0), ("C3", 350, 345, 3, 1), ("C4", 145, 146, 1, 0), ("C5", 275, 210, 3, 0), ("C6", 115, 117, 2, 0)]),
"P07": (True, 72, [("C1", 220, 222, 3, 0), ("C2", 90, 91, 2, 0), ("C3", 310, 305, 4, 0), ("C4", 150, None, 1, 0), ("C5", 250, 251, 4, 0), ("C6", 105, 104, 2, 0)]),
"P08": (True, 88, [("C1", 250, 248, 3, 0), ("C2", 120, 119, 2, 0), ("C3", 380, 330, 3, 1), ("C4", 155, 154, 1, 0), ("C5", 280, 279, 3, 0), ("C6", 110, 108, 2, 0)]),
"P09": (True, 70, [("C1", 190, 189, 3, 0), ("C2", 80, 82, 2, 0), ("C3", 300, 302, 4, 0), ("C4", 140, 140, 1, 0), ("C5", 240, 239, 3, 0), ("C6", 95, 96, 2, 0)]),
"P10": (True, 84, [("C1", 230, None, 3, 0), ("C2", 105, 107, None, 0), ("C3", 360, 355, 4, 1), ("C4", 150, 149, 1, 0), ("C5", 290, 230, 3, 0), ("C6", 100, 101, 2, 0)]),
}
CRITERIA = {"completion": 0.80, "completeness": 0.90, "timing_agreement": 0.80, "max_unclear_per_case": 3, "max_minutes": 90}
done = {p: s for p, s in SESSIONS.items() if s[0]}
completion = len(done) / len(SESSIONS)
fields = [v for _, _, rows in done.values() for r in rows for v in r[1:4]]
completeness = sum(v is not None for v in fields) / len(fields)
pairs = [(r[1], r[2], r[0]) for _, _, rows in done.values() for r in rows if r[1] is not None and r[2] is not None]
agree = sum(abs(a - b) <= 30 for a, b, _ in pairs) / len(pairs)
unclear = {}
for _, _, rows in SESSIONS.values():
for r in rows:
unclear[r[0]] = unclear.get(r[0], 0) + r[4]
over = {p: s[1] for p, s in done.items() if s[1] > CRITERIA["max_minutes"]}
def verdict(ok, near): return "GO" if ok else ("AMEND" if near else "STOP")
print(f"completion {len(done)}/{len(SESSIONS)} = {completion:.2f} (>= {CRITERIA['completion']:.2f}) {verdict(completion >= 0.80, completion >= 0.60)}")
print(f"field completeness {sum(v is not None for v in fields)}/{len(fields)} = {completeness:.3f} (>= 0.90) {verdict(completeness >= 0.90, completeness >= 0.80)}")
print(f"timing agreement {sum(abs(a - b) <= 30 for a, b, _ in pairs)}/{len(pairs)} pairs within 30 s = {agree:.3f} (>= 0.80) {verdict(agree >= 0.80, agree >= 0.60)}")
bad = sorted({c for a, b, c in pairs if abs(a - b) > 30})
print(f" disagreements concentrated in cases: {', '.join(bad)}")
print(f"unclear flags " + " ".join(f"{c}:{n}" for c, n in sorted(unclear.items())) + f" (<= 3 per case) {verdict(max(unclear.values()) <= 3, max(unclear.values()) <= 5)}")
print(f"session length over 90 min: {', '.join(f'{p} ({m})' for p, m in over.items()) or 'none'} {verdict(not over, len(over) <= 2)}")
miss_by_field = [sum(r[i] is None for _, _, rows in done.values() for r in rows) for i in (1, 2, 3)]
print(f"blank fields among completed sessions: observer-1 time {miss_by_field[0]}, observer-2 time {miss_by_field[1]}, severity {miss_by_field[2]}")
Expected output:
completion 9/10 = 0.90 (>= 0.80) GO
field completeness 157/162 = 0.969 (>= 0.90) GO
timing agreement 44/50 pairs within 30 s = 0.880 (>= 0.80) GO
disagreements concentrated in cases: C3, C5
unclear flags C1:0 C2:0 C3:7 C4:0 C5:1 C6:0 (<= 3 per case) STOP
session length over 90 min: P05 (96) AMEND
blank fields among completed sessions: observer-1 time 0, observer-2 time 4, severity 1
How to read it. Completion (9 of 10) and field completeness (157 of 162 fields in completed sessions) are green. Timing agreement is green overall (44 of 50 complete pairs within 30 seconds), but every disagreement is in cases C3 and C5 — the two cases where a participant could reasonably "decide" at more than one moment (C3 invites a provisional rating, then revision after reading the asset note; C5 has a late inject). That is a measurement problem, not an observer problem: the observer form does not define which decision moment to time. Case C3 is flagged unclear by seven participants: red at case level, so C3 cannot be used unrevised. One session ran over 90 minutes (P05, 96), which is amber. The blank fields are not random: four of the five are observer 2's time, which suggests observer 2's form layout (the time box sits on a second page) rather than inattention. P04's incomplete session is excluded from completeness figures by rule but reported, with the reason the participant gave (a work call), in the pilot report.
5.7 Worked example: the pilot report and protocol revision (synthetic)
Decision: amend, then go. The pilot report lists each criterion, the result, and the change made.
Lesson table; scroll horizontally to read all columns →
| Finding | Change | Deviation log entry |
|---|---|---|
| Timing disagreements concentrated in C3, C5 | Define the timed decision as the first written severity on the answer sheet; revisions are recorded separately as a secondary measure | Instrument v1.1, dated, before main study |
| C3 flagged unclear by 7 of 10 | Rewrite C3's asset note so impact is stated, not implied; re-rate with the panel; main-study cases checked for the same ambiguity | Case revision, panel re-rating recorded |
| Observer 2 left time blank 4 times | Move both time fields to the first page; add a completeness check by the facilitator before the next case | Observer form v1.1 |
| One session 96 min | Reduce main-study sessions to 12 cases per period with a 10-minute break; total under 90 minutes in pilot timing | Ethics amendment submitted (session structure) |
| P04 stopped for work reasons | Book sessions outside on-call duties; add a sentence on rescheduling to the information sheet | Information sheet v1.1 |
The report states explicitly: "No effect of the rubric was estimated from pilot data. Ratings were checked only for recording completeness." The main study starts after the ethics amendment is approved.
5.8 Protocol deviations and instrument revisions
A protocol deviation is any departure from the approved protocol: a session run with one observer, a participant who used the rubric sheet in a usual session, a case shown out of order. Record each with date, participant code, what happened, why, and whether it affects the primary outcome. A deviation is not a failure; an unrecorded deviation is. Instrument revisions get version numbers, and every datum records the instrument version that produced it. Changes made after main-study data have been seen require particular care: log whether the change could have been influenced by the data, and if so treat any analysis affected by it as exploratory.
5.9 Stop rules for the main study
Even a well-piloted study needs stop rules: conditions under which the main study is paused or ended. For this study they are: (1) a participant reports distress attributable to the session (pause and review with the ethics committee); (2) evidence of case leakage between participants (pause, investigate, possibly replace cases); (3) a fidelity failure rate above 10% of sessions (pause and retrain facilitators); (4) recruitment below six participants after two months of the recruitment window (consult supervisor; extend or report as an under-powered study with precision stated). There is no stop rule for "the effect looks small" — interim looks at the outcome are not planned and would compromise the analysis.
5.10 Common errors and limits
- Treating the proposal as a formality and leaving partners, ethics and examiners with different understandings of the study.
- Planning around the analysis and discovering that approvals and recruitment were the critical path.
- Writing a risk register without triggers or decided responses.
- Estimating the effect from the pilot, or using it to decide whether to continue.
- Setting progression criteria after seeing pilot results.
- Treating observer disagreement as carelessness when it reveals an undefined measurement moment.
- Revising instruments without version numbers, so that main-study data cannot be linked to the instrument that produced them.
The limit of a pilot is that it is small. It can reveal that something does not work; it cannot prove that everything will.
Exercises — chapter 5
EX027 — Proposal summary. Objective: write the contract. Inputs: the section table in §5.1 and your answers to earlier exercises. Constraints: one page; every section present in one or two sentences. Steps: write it for the Triage Rubric Study, ending with a sentence on what will be reported if the main study recruits only 16 analysts. Expected evidence: the summary. Rubric: 3 = all sections present and consistent with the estimand, design and rivals; fallback stated honestly; 2 = one section vague; 1 = background-heavy; 0 = none.
EX028 — Critical path and risk register. Objective: protect the schedule. Inputs: §5.2–5.3. Constraints: add two risks not in the table. Steps: (1) identify the critical path and slack; (2) add two risks with observable triggers and decided responses; (3) say what happens to the analysis plan if the schedule slips by two months. Expected evidence: a timeline and table. Rubric: 3 = critical path correct (ethics → cases/panel → pilot → recruitment), new risks realistic (for example, a key panel member leaves; a rubric revision is requested by a partner), plan unchanged by slippage; 2 = triggers vague; 1 = generic; 0 = none.
EX029 — Run and modify Lab 5. Objective: audit pilot data and see how criteria respond. Inputs: Lab 5. Constraints: offline. Steps: (1) run it; (2) suppose observer 2 had recorded every blank time correctly, with the value equal to observer 1's: edit the four blanks in completed sessions accordingly and rerun; (3) report how completeness and timing agreement change; (4) explain why the C3/C5 conclusion does not change. Expected evidence: outputs and a paragraph. Rubric: 3 = completeness rises to 161/162, timing pairs rise to 54 with 48 within 30 s (0.889), the C3/C5 diagnosis unchanged and explained; 2 = one error; 1 = not run; 0 = none.
EX030 — Progression criteria for a different pilot. Objective: derive thresholds from main-study needs. Inputs: a pilot of a playbook-format study where participants must complete a 20-item knowledge test before and after. Constraints: at least four criteria. Steps: for each criterion, give green/amber/red thresholds and the main-study requirement that justifies each green threshold. Expected evidence: a table. Rubric: 3 = thresholds justified by main-study needs (for example, completion needed for precision; item completeness needed for scoring), case- or item-level stops distinguished from study-level; 2 = thresholds arbitrary; 1 = criteria without thresholds; 0 = none.
EX031 — Deviation log. Objective: record deviations properly. Inputs: three events in the main study: (a) session A09 ran with one observer; (b) analyst A14 kept the rubric sheet on the desk during the usual period and said they "glanced at it twice"; (c) case C19 was shown before C18 to analyst A20. Constraints: use the fields in §5.8. Steps: write the three entries and, for each, say whether and how it affects the primary analysis. Expected evidence: the log and a paragraph. Rubric: 3 = entries complete, (b) recognized as a fidelity failure affecting the primary outcome with a predeclared sensitivity analysis, (a) and (c) judged minor with reasons; 2 = one judgement unsupported; 1 = entries incomplete; 0 = none.
EX032 — The pilot report paragraph. Objective: report a pilot without estimating an effect. Inputs: Lab 5 output and §5.7. Constraints: 200 words; no rubric effect estimated. Steps: write the report's results paragraph and its decision statement. Expected evidence: the paragraph. Rubric: 3 = each criterion reported with numerator, denominator and verdict, measurement diagnosis explained, decision linked to changes; explicit statement that no effect was estimated; 2 = verdicts without numbers; 1 = effect discussed; 0 = none.
Chapter synthesis
A proposal is a contract: it fixes the question, design, measures, rivals, ethics and outputs before data. Plan around the critical path of approvals, materials and recruitment, and keep a risk register with observable triggers. A pilot tests feasibility and measurement, never the effect. Predeclare progression criteria with go, amend and stop thresholds, audit the pilot against them, and read disagreement and missingness as diagnostics of the instruments. Record every deviation and revision with versions and dates, and set stop rules that protect participants and data, not the hoped-for result.