◇ OpenCyber Academy
Master's · Chapter 8 · English

Chapter 8 — Independent review and defense

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 8 — Independent review and defense

Lesson map

Lesson 08 published. Map, example, EX045.

flowchart TD
  N0["Learning outcomes"]
  N1["8.1 Examination"]
  N2["8.2 Questions expected"]
  N3["8.3 The defence"]
  N4["8.4 Revisions"]
  N5["8.5 Sharing results"]
  N6["8.6 Follow-through"]
  N7["Lessons for practice"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX045"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["8.1 Examination"]
  N2["8.2 Questions expected"]
  N3["8.3 The defence"]
  N4["8.4 Revisions"]
  N5["8.5 Sharing results"]
  N6["8.6 Follow-through"]
  N7["Lessons for practice"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX045"]

[Learning outcomes]
    |
    v
[8.1 Examination]
    |
    v
[8.2 Questions expected]
    |
    v
[8.3 The defence]
    |
    v
[8.4 Revisions]
    |
    v
[8.5 Sharing results]
    |
    v
[8.6 Follow-through]
    |
    v
[Lessons for practice]
    |
    v
[Worked example] --> [Exercise EX045]

Learning outcomes

You will (1) prepare for independent examination; (2) respond to examiner questions; (3) revise the dissertation; (4) share results responsibly.

8.1 Examination

An external examiner reads the dissertation and the archive. Lucas prepares by rerunning all scripts from the archive on a clean machine. Rerunning on a clean machine checks that the archive is complete and that the readme's instructions work for someone other than Lucas. Any difference between the rerun and the dissertation's tables would be found before the examiner finds it. He records the rerun date and the matching outputs in the archive. He also rereads the pre-registration, so he can say which choices were fixed before the data arrived.

8.2 Questions expected

Could origin labels be wrong? He shows the logging checks and mobile-app rule. Is chi-square appropriate? Expected counts exceed five in all cells. Do findings generalise? Mechanisms may, rates may not. Each answer points to evidence in the dissertation: the pilot's labelling checks, the expected-count check before the chi-square test, and the limitations chapter on generalisation. Preparing them in advance helps him answer briefly and accurately under pressure. He also prepares an answer on the register: who approves intended public resources, and what happens if it is incomplete.

8.3 The defence

In a 45-minute viva, the examiner asks why Lucas did not use multilevel models for team effects. Lucas explains the pre-registered approach and shows the subgroup checks, agreeing that multilevel models would be a good extension. Lucas does not defend the simpler method as better; he defends it as the one he committed to before seeing the data, and shows that the subgroup checks address the concern for now. Agreeing that a multilevel model would be a good extension is honest and costs him nothing.

8.4 Revisions

Minor corrections: clarify the register definition, add the power estimate to the results chapter. The register definition is clarified to say who approves an intended public resource and where approvals are recorded. The power estimate shows how large a difference the study could detect with 1,200 IaC and 600 console changes, which answers the examiner's question about the sample size. Both corrections are made in the archive version too. The archive manifest is regenerated after the corrections, so the digests match the corrected files.

8.5 Sharing results

Larkspur receives a summary with the recommendations; a short paper is submitted with synthetic data and scripts; no engineer or team names are published. The summary for Larkspur leads with the three explanations and their remedies, followed by the limits. The paper uses the synthetic dataset so that other researchers can run the analysis without Larkspur's records. Merged small teams and pseudonyms protect individuals in both, as the ethics application required. The data agreement required the summary to go to Larkspur's data protection officer before wider release.

8.6 Follow-through

Larkspur enables account-level public access blocking and fixes the two templates within a quarter. These are the two recommendations most directly supported by the data, and Larkspur chose to act on them first. Reducing the number of storage administrators takes longer because roles must be reviewed with their owners, as the seminar discussion about unused permissions anticipated. The later repeat measurement used Lucas's scripts unchanged. The repeat measurement is the test that Lucas's revised wording proposed, and its before-after design has the limits he noted.

Lessons for practice

Lucas writes a two-page note for practitioners: measure exposure by change origin, look for template clustering, count identities with storage admin rights, and use provider account-level controls to make private the default. The note avoids naming Larkspur and links to the shared responsibility pages and CIS Benchmarks instead of recommending products. It is shared on the university's practitioner network and later used in a cloud governance workshop.

Archive updates

After corrections, the archive gets a new version and manifest, recorded with the date.

Worked example

Six months later Larkspur repeats the measurement with Lucas's scripts: console exposure has fallen to 1.2% and IaC to 0.3% in synthetic follow-up data, supporting the recommendations while Lucas notes the before-after design's limits.

Synthetic case. Examination and outcomes are synthetic.

Errors and limits

A before-after improvement may reflect other changes too; note them.

Exercises

Synthesis

Prepare reproducibly, answer openly and share responsibly.

Continue studying — chapter 08 of 08

← Chapter 07 (draft) · Course chapter index · Worked results ↓ · Next course via chapter index →