OpenCyber Academy
Doctoral research · Chapter 4 · English
Course code: OCA-193Chapter 04: OCA-193-C004

Chapter 4 — Baseline replication and pilot

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 04 published. Map, example, EX021.

flowchart TD
  N0["Learning outcomes"]
  N1["4.1 Why a baseline"]
  N2["Lab — Baseline profile"]
  N3["4.2 Reading the baseline"]
  N4["4.3 Piloting the checklist"]
  N5["4.4 Adjustments"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX021"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["4.1 Why a baseline"]
  N2["Lab — Baseline profile"]
  N3["4.2 Reading the baseline"]
  N4["4.3 Piloting the checklist"]
  N5["4.4 Adjustments"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX021"]

[Learning outcomes]
    |
    v
[4.1 Why a baseline]
    |
    v
[Lab — Baseline profile]
    |
    v
[4.2 Reading the baseline]
    |
    v
[4.3 Piloting the checklist]
    |
    v
[4.4 Adjustments]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX021]

Learning outcomes

You will (1) replicate a baseline from historical findings; (2) pilot the checklist; (3) adjust procedures before the main study.

4.1 Why a baseline

Before testing a new procedure, Lucía needs to know what usual review finds. She codes sixty historical findings from four past partner reviews, all of contracts already fixed or retired, using the codebook from chapter one. The baseline tells her which categories dominate high-severity findings and whether projects differ, which informs both the secondary hypothesis and the checklist's emphasis. Replicating a baseline also tests whether her coding works on real-looking material rather than only training examples. She reports the coding agreement on this material too.

Lab — Baseline profile

"""Baseline profile of synthetic historical review findings by category and severity (read-only)."""
from collections import Counter
# (project, category, severity) for 60 synthetic findings from past reviews
ROWS = []
for proj, spec in {
    "P1": [("access", "high", 4), ("external-call", "high", 2), ("arithmetic", "medium", 3), ("other", "low", 6)],
    "P2": [("access", "high", 3), ("external-call", "high", 3), ("arithmetic", "medium", 2), ("other", "low", 7)],
    "P3": [("access", "medium", 5), ("external-call", "high", 1), ("arithmetic", "high", 2), ("other", "low", 7)],
    "P4": [("access", "high", 2), ("external-call", "medium", 4), ("arithmetic", "low", 3), ("other", "low", 6)],
}.items():
    for cat, sev, n in spec:
        ROWS += [(proj, cat, sev)] * n

def main():
    print(f"findings {len(ROWS)} across {len({r[0] for r in ROWS})} projects")
    high = [r for r in ROWS if r[2] == "high"]
    print(f"high severity {len(high)} ({100 * len(high) / len(ROWS):.1f}%)")
    for cat, n in Counter(r[1] for r in high).most_common():
        print(f"  high {cat}: {n}")
    for p, n in sorted(Counter(r[0] for r in high).items()):
        print(f"  {p} high: {n}")

if __name__ == "__main__":
    main()

Run with python3 lab_baseline.py. Verified output:

findings 60 across 4 projects
high severity 17 (28.3%)
  high access: 9
  high external-call: 6
  high arithmetic: 2
  P1 high: 6
  P2 high: 6
  P3 high: 3
  P4 high: 2

4.2 Reading the baseline

The run reports sixty findings across four projects, seventeen of them high severity, 28.3 percent. Access control accounts for nine high findings, external calls six and arithmetic two. Projects P1 and P2 have six high findings each, P3 three and P4 two. Lucía notes that access control dominates, consistent with her secondary hypothesis, and that projects differ enough that she must stratify or account for project in the main study rather than pool everything naively. These figures are recorded with the run date and seed-free script.

4.3 Piloting the checklist

Lucía pilots the checklist with two auditors on four contracts not used in the main study. The checklist has thirty-two items grouped by category, each linked to an EthTrust requirement or SWC entry and phrased as a question about the code's intended behaviour. The pilot shows that reviewers skip items they judge irrelevant without recording why, and that three items are ambiguous. Pilots exist to find these problems while they are cheap to fix. Each problem is noted with the item and reviewer comment.

4.4 Adjustments

Lucía adds a not-applicable option that requires a one-line reason, rewrites the three ambiguous items and adds a short section on documentation of intended behaviour, because reviewers struggled to judge correctness without it. She records each change in a deviation log against the registered plan, noting that the pilot was planned and the changes precede main data collection. Supervisors approve the revised checklist, and the main study can begin with a tool that reviewers understand consistently. The revised version is frozen for the main study.

Worked example

The baseline shows 17 of 60 findings are high severity, with access control the largest category; the pilot leads to three rewritten items, a recorded not-applicable option and a documentation section.

Synthetic case. Findings, projects and the checklist are synthetic; EEA EthTrust and the SWC registry are real sources.

Errors and limits

Skipping a pilot leaves ambiguities that add noise to the main study.

Exercises

Synthesis

A replicated baseline and a candid pilot ground the main study in real patterns and a usable instrument.

Continue studying — chapter 04 of 08

← Chapter 03 (draft) · Course chapter index · Chapter 05 (draft) →

Worked answers

EX021. Sixty historical findings from four past reviews of fixed or retired contracts.

EX022. 17 of 60, 28.3 percent.

EX023. Access control, with nine.

EX024. Projects differ, from six high findings to two.

EX025. Reviewers skipped items without reasons and three items were ambiguous.

EX026. In a deviation log against the registered plan.