OpenCyber Academy
Master's · Chapter 6 · English
Course code: OCA-174Chapter 06: OCA-174-C006

Chapter 6 — Data collection and analysis

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 06 published. Map, example, EX033.

flowchart TD
  N0["Learning outcomes"]
  N1["6.1 Collection"]
  N2["Lab — exposure by origin"]
  N3["6.2 Primary result"]
  N4["6.3 Subgroups for A"]
  N5["6.4 Clustering for B"]
  N6["6.5 Identity for C"]
  N7["Checking the data"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX033"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["6.1 Collection"]
  N2["Lab — exposure by origin"]
  N3["6.2 Primary result"]
  N4["6.3 Subgroups for A"]
  N5["6.4 Clustering for B"]
  N6["6.5 Identity for C"]
  N7["Checking the data"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX033"]

[Learning outcomes]
    |
    v
[6.1 Collection]
    |
    v
[Lab — exposure by origin]
    |
    v
[6.2 Primary result]
    |
    v
[6.3 Subgroups for A]
    |
    v
[6.4 Clustering for B]
    |
    v
[6.5 Identity for C]
    |
    v
[Checking the data]
    |
    v
[Worked example] --> [Exercise EX033]

Learning outcomes

You will (1) collect the full dataset under the agreed rules; (2) run the pre-registered analysis; (3) test subgroup patterns for each explanation; (4) interpret with care.

6.1 Collection

Larkspur exports one year of storage change records, pseudonymised, with origin, team group, hour, resource and template identifier. Lucas checks counts against Larkspur's monthly totals before analysis. The check against Larkspur's monthly totals guards against a month missing from the export. Team groups follow the ethics rule merging small teams, and origin labels include the mobile-app mapping found in the pilot. Lucas records the export date and the hash of each file in the archive before running the pre-registered scripts.

Lab — exposure by origin

"""Exposure events by change origin: rates and a chi-square test of independence (stdlib only)."""
import math
table = {"iac": (12, 1188), "console": (27, 573)}  # (exposing changes, other changes)
for k, (e, o) in table.items():
    print(f"{k:8} {e}/{e+o} changes exposing = {100*e/(e+o):.1f}%")
rows = list(table.values()); N = sum(sum(r) for r in rows)
cols = [sum(r[i] for r in rows) for i in (0, 1)]
chi = sum((rows[i][j] - sum(rows[i]) * cols[j] / N) ** 2 / (sum(rows[i]) * cols[j] / N) for i in (0, 1) for j in (0, 1))
p = math.erfc(math.sqrt(chi / 2))  # 1 degree of freedom
rr = (27/600) / (12/1200)
print(f"chi-square {chi:.1f} (df 1), p {p:.2g}; risk ratio console/iac {rr:.1f}")

Run with python3 lab_origin.py. Verified output:

iac      12/1200 changes exposing = 1.0%
console  27/600 changes exposing = 4.5%
chi-square 23.1 (df 1), p 1.5e-06; risk ratio console/iac 4.5

6.2 Primary result

In the synthetic dataset, 12 of 1,200 IaC changes (1.0%) and 27 of 600 console changes (4.5%) caused unintended public exposure. The chi-square statistic is 23.1 with one degree of freedom, p about 1.5 × 10⁻⁶, and the risk ratio is 4.5: console changes were four and a half times as likely to expose storage. The result supports explanation A's main prediction, but on its own it cannot show why console changes are riskier. That requires the subgroup checks and the tests of B and C. Counts sit beside each rate, so readers can recheck them.

6.3 Subgroups for A

The difference holds within daytime and night-time changes and within the three largest team groups, which supports explanation A beyond timing or team effects. Lucas reports the rate in each subgroup with its counts, so readers can see that the difference is not driven by one large team. Smaller teams merged under the ethics rule are reported as one group. Identity breadth, added as a control, is examined with explanation C. The subgroup tables are in an appendix with their counts.

6.4 Clustering for B

Nine of the twelve IaC exposures come from two templates, supporting explanation B: weak defaults spread. Fixing those templates would address most IaC exposures. The two templates were copied into several projects, carrying the same permissive storage setting each time. The remaining three IaC exposures came from other templates, which is consistent with B explaining most but not all IaC exposures. Template identifiers make the fix concrete: Larkspur knows exactly which files to change. Larkspur's platform team uses the template identifiers to find every copy of the permissive setting before changing the source template.

6.5 Identity for C

Accounts where more than ten identities hold storage administration rights have higher exposure rates for both origins, supporting C as a contributing factor. The pattern holds for both origins, which means identity breadth is not simply a disguise for console use. Lucas reports C as a contributing factor rather than the main cause, because the study observes accounts at one organisation and cannot separate breadth from other account differences. The threshold of ten identities was chosen before analysis. Breadth counts per account are in the appendix, so readers can see how many accounts sit above the ten-identity threshold.

Checking the data

Before analysis, Lucas checks for duplicate records, missing origins and impossible timestamps. He finds 14 records with no origin, all from an automation account; after checking with Larkspur, they are classed as IaC pipeline changes and the rule is logged. A sensitivity analysis excluding them gives the same conclusion.

Worked example

Lucas's conclusion is that all three explanations contribute, with different remedies: guardrails for console changes, template fixes for IaC and fewer storage administrators. Dr Tanaka praises the result for replacing an argument between teams with evidence that both were partly right.

Synthetic case. All counts are synthetic.

Errors and limits

A small p-value shows the difference is unlikely by chance, not that it is large or causal.

Exercises

Synthesis

Test each explanation's prediction and match remedies to causes.

Continue studying — chapter 06 of 08

← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →

Worked answers

EX033. IaC 1.0% (12/1,200), console 4.5% (27/600).

EX034. Chi-square 23.1 (df 1), risk ratio 4.5.

EX035. Nine of twelve.

EX036. Accounts with over ten storage admins had higher rates for both origins.

EX037. Console: guardrails; IaC: template fixes; identity: fewer storage admins.

EX038. The difference is unlikely by chance, not that it is large or causal.