Chapter 3 — Original contribution proposal
Lesson map
Lesson 03 published. Map, example, EX015.
flowchart TD
N0["Learning outcomes"]
N1["3.1 Hypotheses and measures"]
N2["3.2 Defining the intervention"]
N3["3.3 Sample size and effect"]
N4["3.4 Threats to validity"]
N5["3.5 Review and approval"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX015"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["3.1 Hypotheses and measures"]
N2["3.2 Defining the intervention"]
N3["3.3 Sample size and effect"]
N4["3.4 Threats to validity"]
N5["3.5 Review and approval"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX015"]
[Learning outcomes]
|
v
[3.1 Hypotheses and measures]
|
v
[3.2 Defining the intervention]
|
v
[3.3 Sample size and effect]
|
v
[3.4 Threats to validity]
|
v
[3.5 Review and approval]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX015]
Learning outcomes
You will (1) write a proposal that states hypotheses, measures and analysis in advance; (2) define the intervention precisely enough to replicate; (3) plan sample size against realistic effect sizes; (4) anticipate threats to validity.
3.1 Hypotheses and measures
Amara's proposal states one primary hypothesis: teams that add a STRIDE threat-modelling step to design review will show a lower rate of post-release architecture defects per hundred design changes than comparison teams, measured over twelve months after each change. Secondary hypotheses concern defect severity and time to fix. Each measure has a definition, a data source and a coding rule. She also states what is not measured, such as defects never discovered, and explains why: they cannot be counted, so the study can only estimate effects on discovered defects, a limitation she will repeat in every chapter that reports results.
3.2 Defining the intervention
A vague intervention cannot be replicated. Amara specifies it in detail: before design approval, the team draws a data-flow diagram with trust boundaries, walks each element through the six STRIDE categories, records threats with owners and mitigations, and an architect from outside the team reviews the record. The session is time-boxed at ninety minutes for a typical change. She writes a fidelity checklist so that each session can be scored for whether it followed the protocol. Without fidelity data, a null result would be impossible to interpret, because nobody would know whether the practice was really applied.
3.3 Sample size and effect
Architecture defects are rare: Tessaro's history shows about four per hundred design changes. Amara asks what reduction would matter to Tessaro, and the security leadership says a quarter fewer defects would justify the time cost. With thirty teams and roughly forty design changes per team per year, she simulates the study many times under that assumed effect and finds that detecting it reliably needs at least eighteen months of follow-up. She records the simulation, its assumptions and the resulting plan, and she notes that a smaller true effect could easily go undetected, which she will report rather than hide.
3.4 Threats to validity
Following Wohlin et al. (2012), Amara lists threats in four groups. Construct validity: the defect classification may not capture what architects mean by an architecture flaw. Internal validity: adopting teams might also change other practices, such as adding security champions. External validity: an insurance company with a central architecture function may differ from a startup. Conclusion validity: rare outcomes give wide uncertainty. For each threat she writes a mitigation, such as recording concurrent practice changes every quarter and reporting confidence intervals rather than only significance tests.
3.5 Review and approval
Amara presents the proposal to her committee and to Tessaro's architecture board. The committee asks why she does not also measure threats found in sessions; she adds it as a process measure, not an outcome, because finding more threats does not prove fewer defects. Tessaro asks that teams not be ranked publicly; she agrees to report only aggregates. Both groups sign off, and she registers the proposal with a timestamp. She keeps a change log: any later change to measures or analysis will be recorded with a reason and reported, so readers can distinguish planned from exploratory results.
Worked example
Amara writes a proposal with one primary and two secondary hypotheses, a ninety-minute STRIDE protocol with fidelity checklist, an eighteen-month follow-up based on simulation and a validity table.
Synthetic case. Defect rates, team counts and board decisions are synthetic; Wohlin et al. (2012) and Shostack (2014) are real.
Errors and limits
Simulated power depends on assumed variation between teams. Fidelity checklists can become box-ticking if reviewers do not observe sessions.
Exercises
- EX015 — Undiscovered defects. Why can the study only estimate effects on discovered defects? Rubric: 3 = undiscovered defects cannot be counted, so they are outside the measure;; 2 = cannot count;; 1 = includes all;; 0 = none.
- EX016 — Fidelity purpose. Why is a fidelity checklist essential for interpreting a null result? Rubric: 3 = shows whether the practice was really applied; otherwise null is ambiguous;; 2 = applied;; 1 = not needed;; 0 = none.
- EX017 — Meaningful effect. What reduction did Tessaro leadership say would matter? Rubric: 3 = a quarter fewer architecture defects;; 2 = quarter;; 1 = any;; 0 = none.
- EX018 — Follow-up length. What follow-up did the simulation suggest, and why? Rubric: 3 = at least eighteen months, because defects are rare;; 2 = eighteen months;; 1 = six months;; 0 = none.
- EX019 — Internal threat. Give an internal-validity threat and its mitigation. Rubric: 3 = adopting teams also change other practices; record concurrent changes quarterly;; 2 = threat only;; 1 = external threat;; 0 = none.
- EX020 — Threats-found measure. Why is threats found in sessions a process measure, not an outcome? Rubric: 3 = finding more threats does not prove fewer defects;; 2 = not outcome;; 1 = is outcome;; 0 = none.
Synthesis
A strong proposal fixes hypotheses, intervention, sample plan and validity threats in writing before outcome data are seen.
Continue studying — chapter 03 of 08
← Chapter 02 (draft) · Course chapter index · Chapter 04 (draft) →