Chapter 4 — Baseline replication and pilot | Doctoral research Cryptography and trust services

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 04 published. Map, example, EX021.

flowchart TD
  N0["Learning outcomes"]
  N1["4.1 Why a baseline first"]
  N2["Lab — Baseline audit"]
  N3["4.2 Reading the baseline"]
  N4["4.3 Planning from the baseline"]
  N5["4.4 The pilot"]
  N6["4.5 Revisions before the main study"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX021"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["4.1 Why a baseline first"]
  N2["Lab — Baseline audit"]
  N3["4.2 Reading the baseline"]
  N4["4.3 Planning from the baseline"]
  N5["4.4 The pilot"]
  N6["4.5 Revisions before the main study"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX021"]

[Learning outcomes]
    |
    v
[4.1 Why a baseline first]
    |
    v
[Lab — Baseline audit]
    |
    v
[4.2 Reading the baseline]
    |
    v
[4.3 Planning from the baseline]
    |
    v
[4.4 The pilot]
    |
    v
[4.5 Revisions before the main study]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX021]

Learning outcomes

You will (1) measure a baseline misuse rate from a historical audit; (2) run a pilot that tests procedures rather than effects; (3) revise instruments before the main study.

4.1 Why a baseline first

A baseline tells Nadia how common misuse already is, which shapes power, and it replicates a claim Meridian believes but has never checked: that its wrapper library has made misuse rare. Two auditors, blind to the coming study, review a random sample of 80 crypto-touching changes merged in the previous year, using the Chapter 1 codebook with its new overlap rule. Each confirmed finding is linked to a pseudonymised change identifier. Nothing in the sample is shared outside Meridian until fixes for any open issues have shipped.

Lab — Baseline audit

"""Baseline count of confirmed crypto-misuse in 80 synthetic merged changes (read-only)."""
from collections import Counter
# (change id, confirmed misuse category or None) -- synthetic audit of last year's merges
CHANGES = ([("nonce-iv")] * 7 + [("key-handling")] * 9 + [("primitive-choice")] * 5 +
           [("other")] * 2 + [None] * 57)

def main():
    c = Counter(x for x in CHANGES if x)
    n = len(CHANGES); m = sum(c.values())
    print(f"changes {n}, with confirmed misuse {m} ({100 * m / n:.1f}%)")
    for k, v in c.most_common():
        print(f"  {k:<17} {v}")

if __name__ == "__main__":
    main()

Run with python3 lab_baseline.py. Verified output:

changes 80, with confirmed misuse 23 (28.8%)
  key-handling      9
  nonce-iv          7
  primitive-choice  5
  other             2

4.2 Reading the baseline

Twenty-three of eighty changes, 28.8 percent, contain at least one confirmed misuse. Key-handling leads with nine, followed by nonce-iv with seven, primitive-choice with five and other with two. The replication therefore fails Meridian's belief: the wrapper reduced some algorithm mistakes, but keys still end up in configuration files and nonces are still sometimes derived from timestamps. Nadia records the rate as roughly 2.9 findings per ten changes for planning, while noting that one change can hold several findings and that her primary outcome counts findings, not changes.

4.3 Planning from the baseline

With a control mean near 1.5 to 2 findings per ten changes per team per quarter, Nadia judges that a halving would be detectable with twelve teams per arm but a 20 percent reduction would not. She writes this into an addendum to the registration, timestamped before the main study begins, so examiners can see the planning arithmetic was not adjusted afterwards. She also notes the distribution is skewed with many zeros, another reason the pre-registered permutation test suits the data better than a t-test.

4.4 The pilot

Two volunteer teams, excluded from the main study, use the checklist and lint step for four weeks. The pilot asks procedural questions only. Do reviewers complete the checklist? Does the lint rule for constant nonces fire on wrapper calls? Can the blinded auditor confirm findings without seeing which arm produced them? Reviewers complete 31 of 34 checklists. The nonce rule produces false positives on test fixtures, which use fixed nonces deliberately with throwaway keys. One auditor guesses the arm from checklist comments left in merge descriptions, a blinding leak.

4.5 Revisions before the main study

Nadia makes three changes and logs each as a pre-study revision. The lint rule skips files under test directories, documented so examiners can judge the trade-off. Checklist answers move from merge descriptions to a separate form the auditor never sees, closing the blinding leak. A checklist question about key rotation is reworded after reviewers misread "rotation" as file-system log rotation. She deliberately does not compute an effect from the pilot, because two non-random teams over four weeks cannot estimate one and a tempting number could bias later choices.

Worked example

From the baseline Nadia reports 23 of 80 changes with misuse, key-handling first, and plans for a halving; from the pilot she fixes the test-fixture false positive, the blinding leak and an ambiguous question.

Synthetic note. The 80 audited changes and both pilot teams are invented for teaching.

Errors and limits

A one-year historical sample may under-count misuse that auditors could not see, such as keys set in deployment environments. Pilots test feasibility; reporting pilot effects as evidence is a common error.

Exercises

EX021 — Baseline rate. What share of audited changes contained misuse?
Show marking criteria
Rubric: 3 = 23 of 80, 28.8 percent;; 2 = percentage only;; 1 = count of findings by category;; 0 = none.
EX022 — Leading category. Which category led the baseline and with how many findings?
Show marking criteria
Rubric: 3 = key-handling, 9;; 2 = category only;; 1 = nonce-iv;; 0 = none.
EX023 — Belief tested. Which Meridian belief did the baseline fail to replicate?
Show marking criteria
Rubric: 3 = that the wrapper library made misuse rare;; 2 = wrapper effective;; 1 = generic;; 0 = none.
EX024 — Detectable effect. Which effect size did Nadia judge detectable and which not?
Show marking criteria
Rubric: 3 = halving detectable, 20 percent reduction not, with 12 teams per arm;; 2 = one;; 1 = "large";; 0 = none.
EX025 — Pilot findings. Name the three problems the pilot exposed.
Show marking criteria
Rubric: 3 = test-fixture nonce false positives, blinding leak via merge comments, ambiguous rotation question;; 2 = two;; 1 = one;; 0 = none.
EX026 — No pilot effect. Why does Nadia refuse to estimate an effect from the pilot?
Show marking criteria
Rubric: 3 = two non-random teams over four weeks cannot estimate it and a number could bias choices;; 2 = too small;; 1 = forgot;; 0 = none.

Synthesis

A replicated baseline sets realistic expectations, and a pilot that hunts for procedural faults, not effects, makes the main study trustworthy.

Continue studying — chapter 04 of 08

← Chapter 03 (draft) · Course chapter index · Chapter 05 (draft) →