◇ OpenCyber Academy
Master's · Chapter 3 · English

Chapter 3 — Study design, ethics and reproducibility

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 3 — Study design, ethics and reproducibility

Lesson map

Lesson 03 published. Map, example, EX015.

flowchart TD
  N0["Learning outcomes"]
  N1["3.1 Design"]
  N2["3.2 Why normalise"]
  N3["3.3 Ethics"]
  N4["3.4 Confidentiality of flaws"]
  N5["3.5 Reproducibility"]
  N6["3.6 Measurement plan"]
  N7["Data management"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX015"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["3.1 Design"]
  N2["3.2 Why normalise"]
  N3["3.3 Ethics"]
  N4["3.4 Confidentiality of flaws"]
  N5["3.5 Reproducibility"]
  N6["3.6 Measurement plan"]
  N7["Data management"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX015"]

[Learning outcomes]
    |
    v
[3.1 Design]
    |
    v
[3.2 Why normalise]
    |
    v
[3.3 Ethics]
    |
    v
[3.4 Confidentiality of flaws]
    |
    v
[3.5 Reproducibility]
    |
    v
[3.6 Measurement plan]
    |
    v
[Data management]
    |
    v
[Worked example] --> [Exercise EX015]

Learning outcomes

You will (1) design a before-and-after study in a live development team; (2) handle ethics and confidentiality for security findings; (3) plan measurement of findings and exposure to change; (4) make the study reproducible without exposing the company.

3.1 Design

An interrupted time series: six releases before the authorisation tests were introduced and six after. The outcome is access-control findings per release from Tallyline's regular penetration tests and bug reports, normalised by lines of code changed.

Twelve releases give six on each side of the change, enough to see whether findings fall at the point the tests were introduced rather than drifting down throughout. Using Tallyline's regular penetration tests and bug reports means the outcome was collected the same way before and after, rather than by a new process that might find more or fewer flaws.

3.2 Why normalise

A quiet release with little change will naturally have fewer findings. Findings per thousand changed lines (KLOC) adjusts for this.

The after period in fact changed more code, 52 KLOC against 41, so normalising works against the tests rather than in their favour. Without it, a reader could fairly ask whether quieter releases explained the drop. Using lines changed rather than total code size matters too: Tallyline's codebase grows every release, but findings come mostly from new or modified code. A KLOC-changed denominator ties the rate to the work that could introduce a flaw.

3.3 Ethics

Developers are not being evaluated; findings are attributed to releases, not people. Tallyline's legal team approves the use of anonymised finding counts. No real vulnerability details leave the company; the dissertation uses synthetic examples.

The ethics committee also asked whether developers might feel blamed if findings rose. Sofia added a statement to the participant information sheet and agreed with the engineering manager that results would not be used in performance reviews. Attributing findings to releases makes that promise easy to keep, because no table has a column for people.

3.4 Confidentiality of flaws

Unfixed flaws are never described in the dissertation. Every finding counted is fixed before publication.

In Tallyline's case that is straightforward, because the study counts findings from past releases, all of which were fixed. The dissertation's examples, such as a customer reading another business's invoice, are synthetic versions written to show the kind of flaw without the real endpoint, parameters or customer. The same rule covers the pilot. The two mismatches the role-matrix tests found, a customer reading another business's invoice and support issuing refunds, were fixed within a day, before any write-up. If a finding were still open at submission, it would be counted in the tables but left out of the text until fixed.

3.5 Reproducibility

Sofia publishes her analysis scripts and a synthetic dataset with the same structure, so others can run the analysis without Tallyline's data.

The synthetic dataset mirrors Tallyline's fields: release number, access-control findings, lines changed, tester-days and notable events. An examiner can run every script on it and see each table produced, even though the figures differ from Tallyline's. After the defence, an examiner asked Sofia to share the dataset with a course on secure development, which she did under CC0.

3.6 Measurement plan

Two people classify each finding as access control or not, using the OWASP Top 10 A01 definition, with disagreements discussed and logged.

Using the A01 definition keeps the classification tied to a public standard rather than Sofia's judgement alone. The disagreement log matters for the analysis: if the two classifiers disagreed often about borderline findings, the before-and-after counts could shift depending on who classified them. In the main data they agreed on 46 of 48. Sofia logs each disagreement with both classifiers' reasons, so a reader can see which findings were borderline and check whether moving them would change the before-and-after counts.

Data management

Finding counts are stored in a university-encrypted folder, separate from any company system, and deleted after the dissertation is examined. Sofia records each data item, where it came from and who approved its use.

Pre-registration

The analysis is registered before the after period begins.

Worked example

The ethics committee asks whether developers might feel blamed if findings rise. Sofia adds a statement to the participant information sheet and agrees with the engineering manager that results will not be used in performance reviews.

Synthetic case. The design and approvals are synthetic.

Errors and limits

A before-and-after design cannot rule out other changes happening at the same time; chapter 4 addresses this.

Exercises

Synthesis

Normalise outcomes, protect people and flaws, and share reproducible synthetic data.

Continue studying — chapter 03 of 08

← Chapter 02 (draft) · Course chapter index · Chapter 04 (draft) →