OpenCyber AcademyLearning outcomes
You will (1) state a falsifiable research question about an architecture practice; (2) choose methods that can tell the answer apart from its rivals; (3) set integrity rules for data, partners and authorship before any data exists; (4) build a coding scheme with measured agreement.
1.1 The candidate and the question
Amara Lind is a doctoral candidate at the fictional Halden Institute of Technology, supervised by Prof. Rafael Quintero. Her partner organisation is the fictional Tessaro Insurance Group, which runs thirty product teams building claims, policy and broker systems. Every person, organisation and number in this book is synthetic. Amara's question is whether a structured threat-modelling step at design review, using STRIDE as described by Shostack (2014), reduces the rate of architecture-level security defects found after release, compared with design review without that step. The question is falsifiable: a well-run study could show no reduction, and she commits in advance to reporting that result.
1.2 Why architecture claims are hard to test
Herley and van Oorschot (2017) argue that much security writing makes claims that cannot be refuted, such as saying a practice improves security without saying what observation would show it does not. Architecture is especially exposed to this problem because outcomes arrive late, defects are rare and many practices change at once. Amara responds by naming a measurable outcome, architecture-level defects per hundred design changes found in the twelve months after release, and by naming rival explanations in advance: team maturity, system age, regression to the mean after bad years and changes in how defects are reported.
1.3 Methods that separate rivals
Amara compares three designs. A before-and-after comparison inside adopting teams cannot separate the practice from general improvement over time. A randomised trial across teams would be strongest, but Tessaro cannot randomise which teams adopt because adoption follows budget cycles. She chooses a staggered adoption design with comparison teams and a difference-in-differences analysis, following the logic in Wohlin et al. (2012) for software engineering experiments, and pre-registers her primary outcome, covariates and stopping rules. She writes down which result patterns would support each rival explanation, so the analysis can test them rather than explain them away later.
1.4 Integrity rules before data
Before collecting anything, Amara agrees integrity rules with Tessaro and her ethics board. Defect records are pseudonymised so no engineer can be identified or blamed. Tessaro may review drafts for confidential details but cannot veto findings. Code, coding manuals and synthetic replicas of the data will be archived, while real records stay with Tessaro. Authorship follows contribution, and the coding assistant who classifies defects will be acknowledged. She records these rules with a date and a hash of the document in her lab notebook, so later readers can see they were not adjusted after the results were known.
1.5 Coding with measured agreement
Classifying a defect as architecture-level or implementation-level is a judgement, so Amara writes a coding manual with definitions and examples. An architecture defect is one whose fix requires changing a trust boundary, component responsibility or data flow; an implementation defect can be fixed inside one component. She and a second coder independently classify a pilot set and compute Cohen's kappa, interpreting it with the cautious bands of Landis and Koch (1977). The lab below computes kappa on forty synthetic pilot defects. A low value would mean the outcome measure is unreliable and the manual needs revision before any comparison is meaningful.
Lab — Coder agreement on defect class
# Cohen's kappa for two coders classifying 40 synthetic defects (A=architecture, I=implementation).
c1 = "AAAAAAAAAAAAAAAAIIIIIIIIIIIIIIIIIIIIIIII"
c2 = "AAAAAAAAAAAAAAIIIIAIIIIIIIIIIIIIIIIIIAII"
n = len(c1); po = sum(a == b for a, b in zip(c1, c2)) / n
pe = sum((c1.count(k) / n) * (c2.count(k) / n) for k in "AI")
print(f"items={n} observed agreement={po:.3f} expected={pe:.3f} kappa={(po - pe) / (1 - pe):.2f}")
Run with python3 lab_kappa.py. Verified output:
items=40 observed agreement=0.900 expected=0.520 kappa=0.79
Worked example
Amara writes the falsifiable question with its outcome and four rivals, chooses staggered adoption with difference-in-differences, dates and hashes her integrity rules and reaches a kappa of 0.79 on the pilot coding.
Synthetic case. Halden Institute, Tessaro Insurance Group, Amara, Rafael and all data are invented; Shostack (2014), Herley and van Oorschot (2017), Wohlin et al. (2012) and Landis and Koch (1977) are real.
Errors and limits
Kappa depends on class prevalence and can mislead when one class dominates. Pre-registration constrains analysis but does not guarantee honest data collection.
Exercises
- EX001 — Falsifiability test. What observation would refute Amara's hypothesis?
Show marking criteria
Rubric: 3 = no reduction, or an increase, in post-release architecture defects per hundred design changes for adopting teams relative to comparison teams;; 2 = names outcome without comparison;; 1 = says it cannot be refuted;; 0 = none. - EX002 — Herley critique. What problem do Herley and van Oorschot identify in security claims?
Show marking criteria
Rubric: 3 = many claims cannot be refuted because they name no observation that would contradict them;; 2 = unscientific claims;; 1 = vague;; 0 = none. - EX003 — Rejecting before-after. Why is a simple before-and-after design weak here?
Show marking criteria
Rubric: 3 = cannot separate the practice from general improvement or regression to the mean;; 2 = one rival;; 1 = strong design;; 0 = none. - EX004 — No randomisation. Why could Tessaro not randomise adoption?
Show marking criteria
Rubric: 3 = adoption follows budget cycles the researcher cannot control;; 2 = budget;; 1 = ethics only;; 0 = none. - EX005 — Industry veto. What right does Tessaro have over drafts?
Show marking criteria
Rubric: 3 = review for confidential details only, no veto over findings;; 2 = review;; 1 = full veto;; 0 = none. - EX006 — Defect definition. How does the manual define an architecture-level defect?
Show marking criteria
Rubric: 3 = fix requires changing a trust boundary, component responsibility or data flow;; 2 = partial;; 1 = any bug;; 0 = none. - EX007 — Pilot kappa value. What kappa did the lab report and how is it read?
Show marking criteria
Rubric: 3 = 0.79, substantial agreement in Landis and Koch bands;; 2 = value only;; 1 = below 0.4;; 0 = none.
Synthesis
Doctoral architecture research begins with a falsifiable question, a design that can separate rivals, integrity rules fixed in advance and a reliable outcome measure.